Beta testing, user feedback improve Post-Gazette’s website redesign

The Pittsburgh Post-Gazette is behaving like a technology startup as it rebuilds its website for the first time in years.

The company is leaning on scientific research, data-driven decision making and public beta testing to produce a new online home that satisfies users and meets the paper’s goals. What does that look like, exactly?

I talked with Pat Scanlon, director of digital strategy and business development, about how it works and the lessons to be learned.

The beta version of ppgnow.com, open for testing.

Scanlon explained that in the alpha stage — early testing during development — the Post-Gazette formed diverse focus groups of five users each. Staff observed how the users tried to interact with the new site and used eye-tracking technology to map where they focused on pages. It assigned users specific tasks, like “find classifieds” or “find where you could advertise on our site,” and tracked how many steps it took them to get there.

That data informed some design revisions, Scanlon said, like breaking page elements up into scannable groups of three to five items and dissolving the oft-ignored right sidebar into individual ads and modules scattered around the other content.

This UserVoice overlay on the beta site shows popular feedback and offers a form to submit new comments.

Once the major decisions were made and the new site was in decent shape, the paper launched it into a public beta stage. A link at the top of post-gazette.com invites readers to try the new ppgnow.com, where they can see what’s coming and leave feedback through the UserVoice tool hovering on every page.

Other news organizations also are embracing the benefits of public beta testing. Last year, the BBC opened a beta version of its new home page a couple of months before it went live. That process, product manager James Thornett wrote, generated “three hundred comments on the blogs, over twenty thousand responses to our online survey, and over two and a half thousand direct emails.”

Here are some of the key lessons that other news organizations can take from what the Post-Gazette is doing.

Feedback fuels improvements. Opening the site before its final launch lets you invite direct feedback about what users like and dislike. In about a month of beta testing so far, the Post-Gazette has received over 780 comments and 2,700 votes supporting one or another comment, Scanlon said. From those, the paper learned that it needed to increase the font size of breaking news headlines on the home page so they would stand out from the time stamps, and that many people show a distaste for dividing articles into multiple pages. The beta process also brings indirect feedback — the chance to look at analytics for patterns of site usage and see if your new design is meeting its goals.

Don’t overreact to feedback. Some people react negatively to any kind of change. Nothing you do will please them. “Relax,” Scanlon said. “Let the feedback come in, and [do] not react to each individual one. Look for trends.”

Betas ease the audience transition. Rather than flipping a switch one morning and jarring the audience, the Post-Gazette gives readers a couple months to see the new beta site and prepare before it launches in February.

Create departmental cooperation. Within a news organization, department silos are one major obstacle to a successful redesign. Different pockets of people in the technology team, the newsroom, sales, marketing and circulation all need input and understanding. The Post-Gazette relied on an assistant editor in the newsroom to lead decisions about organizing content on the new site, and the person leading the technical process spends a lot of time with the newsroom, sales, marketing and audience departments, Scanlon said. “Nobody is surprised; everybody has had their buy-in, and they all realize that they’re going to win some and they’re going to lose some.”

Defend changes with data. Testing and gathering data from users empowers better decision-marking, Scanlon said. It escapes subjective debates or claims tied to tradition. For example, data supported Scanlon’s decision to move links like “classifieds” out of the main navigation bar and up to the top of the page. “The argument that, ‘We’ve always had it over there and that’s where I expect to see it,’ doesn’t hold any water when you have 1,000 people who have user-tested it and say, ‘I don’t see it there, but I can see it up in the top left-hand corner.’”

Use the infrastructure for ongoing change. Openness, transparency and feedback have benefits beyond just a redesign process. Scanlon said the feedback tool will stay on the new site for a while even after it launches, and there will be ongoing testing and improvements. “We won’t do another big redesign like this ever again,” he said. “We will continue this process and make those changes all the way along, rather than one big thing every two or three years. This is an iterative process.”

We have made it easy to comment on posts, however we require civility and encourage full names to that end (first initial, last name is OK). Please read our guidelines here before commenting.

  • http://pulse.yahoo.com/_X54HZRMR2FXERNDZLQZBVSYG6M Laura

    Thanks, Brian.

  • http://twitter.com/scanlon_pittPG scanlon_pittPG

    Jeff-Thx 2 u N @Poynter 4 the article! We R trying hard @PittsburghPG 2 re-imagine media in the age of data. #DataFirst#newspapers

  • http://twitter.com/brianprossi Brian P Rossi

    As an appendix to this article, I’ll share some of the process we went through to come up with the new design of post-gazette.com …

    First, we created a mockup in Photoshop of what the site could look like.  This didn’t include all of the content and advertising modules, but it was a pretty good start.  The mockup contained the header as you see it today.  It had a different sub-menu, one more similar to our existing site.  It was horizontal, and very difficult to use once implemented (one of the reasons we changed it.) 

    We actually broke up the advertising rail before doing any user testing at all, because of research done prior showing the right rail to be a traditional “blind spot.”  I refer to what we’ve done now as utilizing a concept I’ve coined “content affinity.”  The concept being if we place two pieces of content that may or may not be contextually relevant, yet are demographically relevant, we can drive users to the advertising content (classifieds for example) rather than always relying on the “above the fold” / right rail mentality.  It makes the user work a little harder to read the page at first, but it also helps combat banner blindness.  We also marked all advertising content as such, where as on the old site we did not.

    After the mock page was created, we brought people in to get initial emotional reactions.  It was very focused on finding out if the user is happy/warmed before even engaging in the content.  I wanted a site that makes people happy even if they are reading less than happy content.  We tested with a Photoshop image, meaning they couldn’t interact with the site.  We brought in groups of different demographics that for the most part came back with a positive reaction to the design / UX concepts.  We also did a less successful round of content grouping.  The content grouping didn’t work out simply because we gave them entirely too many cards to sort.

    The User Experience was designed on an Agile workflow.  We had requirements, but we constantly used feedback to decide if it was working, and immediately made adjustments.  It doesn’t take a lot of people to find your biggest design flaws.  Before we did real user testing, I spent a lot of time on the street with my iPad asking random people to interact with the half baked site.  This proved to be very beneficial in coming up with the first version of the design.

    So we had a starting point.  Then the real work began.  We spent the next period of time building the beta site.  We went through several iterations of the content flow, and design aesthetics.  After we
    launched the beta, we did a second round of user testing.  This time the users’ had a chance to actually interact.

    For the second round of testing we had the pleasure of having a class at Duquesne University assist us (thanks Bill Gibbs).  We gave them a list of user tasks that were important to the business.  They took that list, recorded users accomplishing the tasks, and timed each one.  They also had associated satisfaction rankings, as well as eye tracking videos that correlated with the tasks.  We found that no one was able to locate real estate, because it sat in the upper right corner, below the
    menu, between two ads.  Since then we moved the classified links, killed the icons (since we were told they looked like ads), and centered the leaderboard advertisement.  We also found that to leave feedback, people scroll right to the bottom, and many didn’t see the tab on the right hand side.  We’re going to make that change, we just haven’t yet.  :)

    The class also helped us make good on my promise to my visually impaired Uncle and Aunt that the site would be accessible/complaint for screen readers.

    The feedback tab has been a source of a lot of the other small improvements that were made as mentioned in this article. 

    We’re far from complete, but we’ve made a lot of good progress.  Like Pat said, we are constantly using the feedback loop to make this an iterative process, even once we launch the site. 

    I want to personally thank the entire team that has made this happen, it’s been a long road for all of us.  A lot of people did a lot of things to make this beta come to life.  Just to name a few in no particular order:

    Chris Szekely (UX design)
    David Esaias (UX design)
    Laura Schneiderman (user testing)
    Brandon Sherman (architecture)
    Tim Dunham (architecture)

    And of course the great executive leadership of the Post Gazette.  :)

  • http://twitter.com/brianprossi Brian P Rossi

    As an appendix to this article, I’ll share some of the process we went through to come up with the new design of post-gazette.com …

    First, we created a mockup in Photoshop of what the site could look like.  This didn’t include all of the content and advertising modules, but it was a pretty good start.  The mockup contained the header as you see it today.  It had a different sub-menu, one more similar to our existing site.  It was horizontal, and very difficult to use once implemented (one of the reasons we changed it.) 

    We actually broke up the advertising rail before doing any user testing at all, because of research done prior showing the right rail to be a traditional “blind spot.”  I refer to what we’ve done now as utilizing a concept I’ve coined “content affinity.”  The concept being if we place two pieces of content that may or may not be contextually relevant, yet are demographically relevant, we can drive users to the advertising content (classifieds for example) rather than always relying on the “above the fold” / right rail mentality.  It makes the user work a little harder to read the page at first, but it also helps combat banner blindness.  We also marked all advertising content as such, where as on the old site we did not.

    After the mock page was created, we brought people in to get initial emotional reactions.  It was very focused on finding out if the user is happy/warmed before even engaging in the content.  I wanted a site that makes people happy even if they are reading less than happy content.  We tested with a Photoshop image, meaning they couldn’t interact with the site.  We brought in groups of different demographics that for the most part came back with a positive reaction to the design / UX concepts.  We also did a less successful round of content grouping.  The content grouping didn’t work out simply because we gave them entirely too many cards to sort.

    The User Experience was designed on an Agile workflow.  We had requirements, but we constantly used feedback to decide if it was working, and immediately made adjustments.  It doesn’t take a lot of people to find your biggest design flaws.  Before we did real user testing, I spent a lot of time on the street with my iPad asking random people to interact with the half baked site.  This proved to be very beneficial in coming up with the first version of the design.

    So we had a starting point.  Then the real work began.  We spent the next period of time building the beta site.  We went through several iterations of the content flow, and design aesthetics.  After we
    launched the beta, we did a second round of user testing.  This time the users’ had a chance to actually interact.

    For the second round of testing we had the pleasure of having a class at Duquesne University assist us (thanks Bill Gibbs).  We gave them a list of user tasks that were important to the business.  They took that list, recorded users accomplishing the tasks, and timed each one.  They also had associated satisfaction rankings, as well as eye tracking videos that correlated with the tasks.  We found that no one was able to locate real estate, because it sat in the upper right corner, below the
    menu, between two ads.  Since then we moved the classified links, killed the icons (since we were told they looked like ads), and centered the leaderboard advertisement.  We also found that to leave feedback, people scroll right to the bottom, and many didn’t see the tab on the right hand side.  We’re going to make that change, we just haven’t yet.  :)

    The class also helped us make good on my promise to my visually impaired Uncle and Aunt that the site would be accessible/complaint for screen readers.

    The feedback tab has been a source of a lot of the other small improvements that were made as mentioned in this article. 

    We’re far from complete, but we’ve made a lot of good progress.  Like Pat said, we are constantly using the feedback loop to make this an iterative process, even once we launch the site. 

    I want to personally thank the entire team that has made this happen, it’s been a long road for all of us.  A lot of people did a lot of things to make this beta come to life.  Just to name a few in no particular order:

    Chris Szekely (UX design)
    David Esaias (UX design)
    Laura Schneiderman (user testing)
    Brandon Sherman (architecture)
    Tim Dunham (architecture)

    And of course the great executive leadership of the Post Gazette.  :)

  • Anonymous

    Thanks for your thoughts Sean. In fairness to the P-G, I believe they call them “user groups” and I paraphrased it here as focus groups because I think it’s a more commonly understood concept. The activities for the groups were very research-based — observing their interaction with the site, eyetracking, etc. — as opposed to just asking subjective questions about what they want or how the site makes them feel.

  • Anonymous

    It’s great to hear about a news organization observing users interacting with the early stages of a design, but as a UX professional I cringe when I see the term “focus group” used to describe user research, eye tracking studies and the like. In fact, I cringe whenever I hear “focus group,” period. Focus groups have been and continue to be primarily a tool of marketers and advertisers, and are deeply problematic in my opinion. It warms my heart to see news orgs (and businesses of every stripe) embracing design and UX methods for gaining insights to help them make better products. But let’s not confuse those methods with focus groups, which are (I think) past their expiration date.

  • http://twitter.com/brianprossi Brian P Rossi

    Good article, Jeff.  You did a nice job of capturing the process we created for the new site.  The transparency in the user testing data is something new for newspapers that we’re quite proud of.  Thanks for the write-up.  :)