Interview protocol for discovering organizational structures

This page contains our protocol for semi-structured interviews used in the research phase of discovering organizational structures for managing infrastructure in the context of continuous delivery.

Purpose of this document

To guide the interviewer in the preparation, conduction and analysis of semi-structured interviews in the context of our research (organizational structures for infrastructure management in the context of continuous delivery) following academic guidelines, including Grounded Theory.

We also hope that this document can support other researchers in conducting semi-structured interviews for software engineering research.

Purpose of the interviews

By using Grounded Theory, start the development of a theory addressing the following research questions:

• RQ1 Which organizational patterns are organizations currently adopting?
• RQ2 Are there organizational patterns delivering better results than others?
• RQ3 Which forces drive organizations to take different organizational patterns?
• RQ4 Which are the advantages, challenges, and enablers for each organizational pattern?

The questions were designed to address mainly RQ1 and RQ2, and to start to deal with RQ3 and RQ4. For the publication of a first article, RQ1 and RQ2 were merged as

Which organizational structures are software-producing organizations adopting for managing IT technical teams in a continuous delivery context? And what are the properties of each of these organizational structures?

while RQ3 and RQ4 will be more properly handled in the next phase of our research.

Inputs for this protocol

• Original research proposal (research questions above)
• Appendix “Research Protocol for constructing a Conceptual Framework and Mapping Regional Software Startup Ecosystems” of “A Conceptual Framework for Software Startup Ecosystems: the case of Israel” (a technical report of our research group) “Protocol to be used in the interviews at startups” (a document of our research group)
• Article “The Grounded Theory of Agile Transitions” (ICSE, 2017)
• Interview tips of Ijnet, a website for journalists
• Tips from Daniel Cukier, a former PhD student of our research group who conducted interviews within the startup environment
• Our experience with the brainstorm sessions with specialists
• The chapter “Conducting semi-structured interviews” of the book “Handbook of Practical Program Evaluation” (Adams, 2010).
• Our own experience with the interviews conducted under this protocol (we evolved this protocol with more tips based on this experience).

Method

Use of semi-structured interviews. According to Adams, “Conducted conversationally with one respondent at a time, the semi-structured interviews (SSI) employs a blend of closed - and open - ended questions, often accompanied by follow - up why or how questions. The dialogue can meander around the topics on the agenda — rather than adhering slavishly to verbatim questions as in a standardized survey — and may delve into totally unforeseen issues.”

Target population

People who have (in the present or recent past) involvement with teams within software developer organizations who use continuous delivery of software or who are going through the process of adopting continuous delivery.

Guidelines and tips

Before the interview

• Diversified selection. Organization: size, type, location. Interviewee: gender, position (from developer to marketing).
• Test audio recording before the first interview.
• For online interviews, check the audio recording before each interview.
• Choose a recorder that is as discreet as possible. At this point smartphone are suitable, since people are used to cell phones on the table.
• Avoid scheduling interviews in noisy places.
• When possible, prefer face-to-face interviewing to online interviewing.
• Avoid, if possible, asynchronous text-based interviews.
• Prepare a printed version of the questions of this protocol to assist the interviewer during the interview.
• If you make notes on the same board for different interviewees, use different colors.
• When inviting people, provide a link with a dynamic listing of the interviewers’ available days and times. Try to offer as much as time slots as possible.
• Do not send too many invitations at once. You may fill all your slots and, thus, not have time to analyze between interviews.
• Send a LinkedIn connection invitation to the interviewee. It is an opportunity for the interviewee to know better who is interviewing her.

During the interview

• Always be polite.
• Try to be pleasant. A quick unrelated conversation right before the interview can help. Example: “How long have you lived in the city?” Or “Why did you move to São Paulo?”. Easier to do this when the interview is in person.
• Listen. Avoid talking unnecessarily.
• While the interviewee speaks, do not display strong emotions such as enthusiastic agreement, disagreement or surprise.
• Avoid giving opinion during the interview.
• Do not debate (try to contradict) the respondent’s answers.
• You can demonstrate knowledge to the interviewee, but humbly, without putting yourself as more expert than the interviewee.
• At times, repeat what the interviewee said to show interest and understanding in the message.
• Respect the silence, let the interviewee reflect as much as necessary.
• On the other hand, prepare extra questions to keep the interviewee talking. Or improvise. Extra questions can also lead to unexpected revelations.
• Be open to explore other unforeseen topics. New questions more interesting than those already established may arise.
• Do not drive the interviewee to say what we want to hear. On the contrary, if possible try to make him speak what we do not want to hear.
• If the interviewee dodges the question, after a while try to return to the question with another approach. But do not insist too much. Especially in the first interviews, the avoidance may be a sign that the question is not very good.
• Do not follow the script mechanically. The order of the questions in the protocol don’t need to be followed.
• Questions can be tailored to the interviewee’s profile.
• Make hooks based on what the respondent commented.
• Do not be afraid to ask naive questions.
• Ask for clarification when needed. But avoid saying “what do you mean…?” As it seems that you are scolding the interviewee for not communicating properly.
• If the interview is online: if feasible, video chat; otherwise, at least try to greet the interviewee (and say goodbye) by video; and explain to the respondent that the video will be turned off for connectivity reasons.
• It may be impolite to ask the interviewee to turn on the video; The interviewer should just turn on his own video, which suggests to the interviewee to turn on her video too. If in this case the interviewee does not turn on the video, that’s fine.
• Avoid using the term “DevOps” (broad and ill-defined in the community). But especially in explaining the motivation of the interview it can be difficult to escape from this.
• Avoid very specific terms related to the organizational patterns we already know of (eg, SRE).
• Focus on the interviewee’s latest experiences on her team. Do not ask for an explanation about the organization as a whole. Consider projects only from the recent past.
• Estimated length of interview: 1 hour.
• To calibrate the time, you may want to “test the script” before the first interview or conduct the first interview with someone you already know, who will be more tolerant of any endurance.
• Try to minimize note-taking to focus on the interviewee. Write down points during the conversation that lead to the next questions. Choose a standardized place or a different sheet for this, so you do not forget to ask these questions.
• For online interviews, a good software for audio recording is OBS. When starting the interview, pay attention to the audio capture bars (mic and desktop audio), so you are sure that audio is being recorded.

After each interview

• Do the interview analysis as soon as possible. Advantages: 1) the interview is more “fresh” in memory and 2) we have a more agile (iterative) approach, since 10 analyzed interviews are worth more than 40 interviews without analysis.
• Check the audio. If the recording did not work, try to do the analysis on the same day or as early as possible.

Interview script

Before the interview

• Collect respondent data. Do not spend interview time on this. Search for data, for example, in Linkedin.
  • Name
  • Linkedin
  • E-mail
  • Sex
  • Graduation course
  • Higher degree (no graduated, graduated, master, PhD)
  • Graduation year
  • Role
  • Years in the company
  • Years in the role
  • Company
  • Company founding year
  • Number of employees in the company
  • Number of IT employees in the company
• Research about the interviewee before the interview. This search goes beyond profile data. It helps to create “intimacy” with the interviewee.
• On the interview eve, remind the interviewee about the interview.
• If the interview is in person: arrive early (~ 15 min); if possible try to know the environment of the company.

Starting the interview

• Mention group researchers.
• Explain the purpose of the research.
• Mention expected benefits for the community (software producing organizations).
• Mention that the interview can also benefit the interviewee by helping her to reflect on her work environment.
• Explain the protocol.
• Explain why the recording and its confidentiality.
• Explain that the publications will treat companies and respondents anonymously.
• Explain that any results will be sent to the respondent firsthand.
• Mention that our group will be open for future collaborations.
• Ask permission to start recording.
• Start recording by saying “OK, it’s recording now.”
• Start with an easy and open question (“icebreaker”). Helps the interviewee become more comfortable.

During the interview

• Record the audio.
• Record with the cellphone. But take the portable recorder in case of any problem with the phone.
• Take only a few notes (on the computer) but always remember to keep eye contact with the interviewee.
• If another researcher is present, the second researcher may take further notes.

Questions template

• Question
  • Additional incentive hints and questions for respondent to speak more
  • Rationale
  • Notes on the evolution of the question

Main questions

• Please, tell me about your company, your role within your company, and the project in which are currently working.
  • Icebreaker question
• Who is responsible for deploying the solution in production?
  • Directly linked to RQ1
  • Starting with “relevant but still non threatening questions” (Adams, 2010).
• In a new project, who builds the new environment?
  • Directly linked to RQ1
• And about the Cloud? PaaS / serverless?
  • What do you think?
  • In our DevOps literature review, we already found that the use of cloud platforms, especially PaaS and serverless, can strongly impact on expected skills of software engineers.
  • Usually we did not need to ask this directly.
• Who is in charge for non-functional requirements?
  • Integrity, availability, flow rate, response time.
  • Capacity planning, load balancing, overload management, timeout and retrials configuration.
  • Deploys without unavailability.
  • Directly linked to RQ1
• Who is responsible for configuring monitoring?
  • Directly linked to RQ1
• Who is responsible for tracking the monitoring?
  • Directly linked to RQ1
• Who is on-call for incident handling?
  • After-hours frequency
  • Blameless post-mortem?
  • Directly linked to RQ1
• Delivery performance:
  • Time from commit to production
    • < 1h / 1 week < t < 1 month / > 1 month
  • Deployment frequency
    • Under demand (many times per day)
    • 1 per week < f < 1 per month
    • f > 1 per month
  • Mean time for repair
    • < 1h / < 1 day / 1 day < t < 1 week / > 1 week
  • Frequency of failures (% of deliveries)
    • 0 - 1% / 1 - 15% / 15 - 30% / 30 - 50% / Mais de 50
  • Reasons for the above results.
  • Directly Connected to RQ2. Provides a link between the organizational structure and its “delivery performance” (“better results” in RQ2). We use “delivery performance” as defined by Forsgren. The questions alternatives follow the results found in the State of DevOps Surveys.
• What would you change in this work system of your team/company?
  • Why is it hard to implement such changes?
  • Why did this problem arise? Why do they remain?
  • Directly connected to RQ2. A more qualitative approach to feel the respondent’s (in)satisfaction with the results of the organizational pattern in which she is immersed.
  • Sub-questions help to answer RQ3, because at this point the interviewee will probably justify the reasons for the existing organizational structure.
  • Sub-questions help to answer RQ4 since what could be improved may refer to “advantages, challenges and enablers” of the existing organizational pattern.
  • Follows the guideline “rather than asking people to identify what is “bad,” asking advice on “how to make things better” and “areas that need improvement” can help minimize defensiveness” (Adams, 2010).
  • “The most potentially embarrassing, controversial, or awkward questions should come toward the end” (Adams, 2010). In that case, the most complicated of the main questions is at the end of the main questions.

Specific questions

These questions should be dynamically chosen according to the “advantages, challenges and enablers” that have so far been revealed in the interview. They serve to deepen the understanding of the “advantages, challenges and enablers” of the existing organizational structure.

• Inter-team communication; problem?
  • Little communication? Over communication?
  • The classic idea of DevOps calls for closer approximation between devs and ops (“tearing down walls between silos”). But more communication is not necessarily better communication, especially between different departments. Organizational patterns may be directly linked to the need for communication between the organization’s professionals.
• Are different teams aligned (committed to the project)?
  • How do this happen?
  • Consequences
  • DevOps literature advocates that different teams and departments should be aligned. But little is said about how to achieve such alignment. However, in organizational patterns that maintain separate, collaborating departments (devs and ops), alignment is essential and, therefore, deserves to be questioned.
• Clear definition of responsibilities?</span>
  • If not, any problem?
  • If yes, long hand-offs?
  • There are reports of “DevOps deployments” in which devs and ops passed to share responsibilities, especially on deployment automation. However, reports show that conflicts arise from this responsibility sharing as professionals question the responsibilities of those involved in the project (“Am I doing something the other should do?”, “Are they doing what I should do?”).
• Is there a “DevOps team”? What is it about?
  • The literature comments on the existence of devops teams in organizations, but little is known what exactly these teams are.
• Are there people with the “DevOps role”? What is it about?
  • The literature also comments on the existence of the “devops engineer” or “full-stack engineer”. However, the very initial idea of DevOps (devs and ops collaborating) seems to be against the idea of a DevOps role.
• If there are no “dedicated ops”: how do software engineers improve their skills on infrastructure?
  • Were they hired with such skills?
  • Everyone in the team knows about infrastructure?
  • How people are chosen for “knowing more” about infrastructure.
  • The lack of “exclusive ops” is a feature we expect to find in many organizations. It is associated with the so-called cross-functional teams. Advocates argue for higher productivity (less wait time between teams) in this model. But the big challenge is getting a single team to possess a wide range of expertise in different technical topics.
  • After interview #I20, we diminished the focus on this question and the following two ones since they were not contributing so much for the evolution of our taxonomy.
• Is there any kind of incentive from the company about continuous education?</span>
  • community of practices
  • FLOSS
  • internships
  • teams mutation
  • timeshare for studying
  • Again, in cross-functional teams, the ability to learn is even more important than in more traditional structures.
• If a team needs knowledge/skills that they do not have, what to do?
  • Study with team mate
  • Search for help in another teams
  • Search for in-home specialist
  • Search for external consultant
  • Continuation of the previous question.
• Does each team have their own specialists? Are there shared specialists?
  • Sharing policy
  • Handoffs problems
  • The way specialists spread across the organization is totally tied to the issues of organizational patterns.

Extra questions

Most of these questions further explore the impacts of choices on how experts spread across the organization.

• How does the “DBA role” work in your organization?
• Is there production problems related to database?
  • Availability, connection between application and database
  • By the end, we asked less this specific question.
• Are there any security experts in the company?
• How do developers become aware about security concerns?
  • Ex: known vulnerabilities
• For your last project, were vulnerabilities found in production?
  • Were they exploited by attackers?
  • “The most potentially embarrassing, controversial, or awkward questions should come toward the end (…) along with reminders of confidentiality as needed” (Adams, 2010).
  • By the end, we asked less this specific question.
• Microservices: do you use? What do you think?
  • This question will probably arise naturally if that is the case. But as it is a theme strongly linked to DevOps and cross-functional teams, it is worth to briefly explore it. But it is good to leave more to the end, so it does not improperly take the time of the interview.
  • As we rose some hypotheses related to microservices, we strengthened the relevance of this question. However, most of the times we did not need to ask it directly.

Some other additional questions we created after #I20 to support the elaboration of rising hypotheses:

• Do you have few automated tests? How do you deal with the tradeoff velocity vs quality?
• How is the collaboration among developers and members of the platform team?
• Do you feel the platform provides more benefits for microservices than for monolithic services?

Ending the interview

• Ask: “About your technology stack, would you have a favorite tool to recommend?” Quick and easy research-related question to give a final “chill out”. I think in general people are excited to talk about tools.
  • Due to time, in general this question was not asked.
• Ask: “Would you have any questions for me or a final comment?”
• Ask for suggestions of other people to be interviewed (snowball).
• Ask to take a picture together: “Can I take a selfie with you?”
• After the interview is over, wait a moment: sometimes it is at this time of relaxation that the interviewee remembers something interesting to add. Selfie time can also help in this regard.
• Thank for the interview.
• Offer a business card.
• At the end of the day, send a “thank you” message.
• After the interview, save the audio with metadata: interviewee, date and location.

After each interview

• Analyze what went right and what went wrong in the interview:
  • Have the research questions been answered?
  • Were the guidelines and tips followed?
  • Has any new and exciting topic come up?
  • Raise improvement points for upcoming interviews.
  • If necessary, evolve the interview protocol.
  • Do this exercise mainly for the first interview.
• Take notes of references (books, articles, lectures, etc.) that may have been indicated by the interviewee.

Invitation e-mail

Subject: Collaboration with scientific research about DevOps

Dear XXX,

I’m a Computer Science Ph.D. student at the University São Paulo, advised by Prof. Fabio Kon. We are researching the various ways of organizing teams and departments for managing infrastructure in the context of continuous delivery. And for this purpose, we are interviewing IT professionals.

With your experience as role in organization, you were recommended by YYY as an important professional to contribute to our research. Therefore, we would be pleased to interview you for our study, if you believe your scenario matches our requirements, i.e.: your work relates to some product delivered by a continuous delivery* flow (or if there are changes occurring toward continuous delivery).

* With “continuous delivery” we mean that after developing a system increment, the deployment at production is achieved by the issue of a simple command or the press of a single button. This flow is implemented by which is called the “deployment pipeline”.

We expect our research to help software-producing organizations worldwide in taking more informed decisions about organizing people for managing infrastructure in the context of continuous delivery. The interview can also be useful for you as an exercise of thinking over your current organization status.

Could we schedule a conversation? Something between 45 or 60 minutes would be great. The interviews, as well the interviewees and their organizations, will be anonymized in our publications.

Thank you very much,
Leonardo Leite
Computer science PhD student
University of São Paulo (USP)
www.ime.usp.br/~leofl/en.html