My favorite developer productivity research method that nobody uses

Mihaly Csikszentmihalyi pioneered research on flow state. Wrote popular book Flow: The Psychology of Optimal Experience back in 1990. Lorin Hochstein read it many years ago. One thing stuck around most was research method Csikszentmihalyi used to study flow. One of challenges of studying people's experiences is difficult for researchers to observe them directly. This problem comes up when organization tries to do research on current state of developer productivity within organizations. So much of work we do in software world is so hard for others to see.

Different data collection techniques developer productivity researchers use include surveys, interviews, focus groups, as well as automatic collection of metrics like DORA metrics. Of those only automatic collection of metrics focuses on in-the-moment data and it's very thin type of data at that. Those metrics can't give you any insights into challenges your developers are facing. Hochstein's preferred technique is case study which he tries to apply to incidents. Likes incident case study technique because gives opportunity to go deep into nature of work for specific episode. But incident-as-case-study only works for incidents and while well-done incident case study can shine light on nature of development work there's also lot it will miss.

Csikszentmihalyi used very clever approach developed by his PhD student Suzanne Prescott called experience sampling. He gave participants of his study pagers and would page them at random times. When paged participants would write down information about their experiences in journal in-the-moment. In this way he was able to collect information about subjective experience without problems you get when trying to elicit account retrospectively. Hochstein has never read about anybody trying to use this approach to study developer productivity and thinks that's shame. It's something he's wanted to try himself except that he has not worked in developer productivity space for long long time.

These days he'd probably use Slack rather than pager and journal to randomly reach out to volunteers during study and collect their responses but principle is same. He's long wanted to capture "are you currently banging your head against wall" metric from developers but with experience sampling you could capture "what are you currently banging your head against wall about?" Would this research technique actually work for studying developer productivity issues within organization? Hochstein honestly doesn't know. But he'd love to see someone try.