Data Collection

Collecting good data is essential to developing a vision for digital-age learning. In traditional research, we start with your question(s) in mind, and attempt to find or collect data that answers our question(s). In a this context, however, you collect data we the purpose of developing our questions. Additionally, a good data collection process will include both qualitative and quantitative data, and use a variety of datasets to ensure that they can find multiple data points to validate a conclusion or indicate a need.

• Quantitative data are data in the forms of numbers and statistics. Quantitative data may come in the form of data collected from surveys, research, frequency counts, expenses, locations, etc.
• Qualitative data are data that are not typically analyzed numerically, but rather through trends and patterns that emerge in their collection. Qualitative data may come in the form of interviews, observations, written documents, etc.

Imagine that you are asked to build a garden for your school. You do your research on the vision for the garden and determine that garden will grow fruits and vegetables that can be sold at cost to community citizens who need access to inexpensive fresh food. In this example, you might want to collect data about soil and sunlight conditions around the school. You might also seek out public data sources about the average temperature and rainfall at different times during the school year, what crops could be grown given the conditions, characteristics of an ideal garden, and what supplies you might need.

In a traditional design model, you would go on the Internet using this data and determine what crops would grow in your garden and plant those. Many community gardens have taken this approach and many of them have failed. When researchers go in to try to figure out why, the answer is simple - those crops aren’t what the people who live around the garden eat. They have no experience with these foods, are uncertain how to prepare them, and aren’t used to cooking with them or eating them.

Design thinking works differently. In a design thinking model, a designer would want to go out into the community and talk to the people who will actually consume the fruits and vegetables to find out what they need. This may include asking about their current diet and finding out what fruits and vegetables they know how to cook with. It will also likely involve finding out if this is a service that they would actually need and if there are any potential obstacles to them using it. Maybe they would find out there’s a vegetable that families would like to work with but need some other supports like a cooking class or dietary consultation. There are no “wrong questions”, and it is always possible to collect more data as new questions arise.

Focus Groups

One technique that is often used to collect data is a focus group. A focus group is a small group of one or more stakeholders (one person is an “interview”) that are pulled together for the purpose of researching the needs of that stakeholder group. Groups may be homogeneous (all the same type of stakeholder, i.e. all students) or heterogeneous (different groups of stakeholders together). Focus group participants should be diverse and a focus group should represent the larger population across multiple dimensions. The goal of a moderator in a focus group is to ask a starter question and then probe for follow-up as needed. It’s a best practice to record and transcribe a focus group so that themes can be more easily identified. In a digital-age learning focus group, I usually only have two questions:

• What should school look like in the digital-age in XYZ school/district?
• How does technology support this goal? From there, I will typically see where the conversation goes, and ask probing questions as needed. For a moderator, it’s critical not to bias the focus group or to ask leading questions while probing for deeper responses. After the first focus group, it’s usually okay to follow-up with major points from previous focus groups to look for points of convergence and divergence.

Focus groups can also be longer activities designed to foster deeper input as in this example from New York City Public Schools:

Surveys

Surveys also provide valuable data for developing a vision. The School Technology Needs Assessment (STNA) for Teachers and for students are older (outdated) surveys that provide insight into student needs, comfort, and use of technology and digital tools. The Texas Teacher School Technology and Readiness (STaR) chart is a rubric that teachers can use to self-assess their readiness and status in digital learning and the Florida Technology Integration Matrix Tools are other resources for data collection as well.

Rubrics

The North Carolina Digital Learning Progress Rubric and the assessments linked in the Future Ready Framework can be used to assess school and district-level readiness for digital-age learning. ISTE also has a document of essential conditions for leveraging technology for learning.

Data Analysis

Once data has been collected, the analysis phase begins. During the analysis phase, we begin to process all of the data that you collected.

As a <persona>, I <want to>, <so that>.

The above prompt is a tool from software engineering, known as a user story. In designing software, programmers will develop hundreds of these stories, focused on the user and their why so that they can ensure a tool will do what their audience needs it to do. We can use a similar protocol when doing data analysis. Keeping our stakeholders and their needs forefront, we can see easily using this protocol if we have overlooked any needs or any audiences in our data collection. We may decide need more data - it’s okay to go collect more. Once we’ve developed our user stories, we can start to do deeper data analysis to lend support and resources for each story.

Begin by reviewing all of the qualitative data collected and look for themes in the data. In the process, use a variety of statistical techniques to organize and review your quantitative data. While this can involve more advanced statistics, you can often get what you need from looking at a simple correlation between two datasets.

The most challenging part of this process is often in identifying how to link two unrelated data sets (e.g., average temperature and average rainfall) to a third dataset (e.g., ideal crops for a specific location). This will be done differently depending on the data collected, but the simplest process is typically the most ideal. The result will likely be a data set that is incomplete, and you’ll find yourself needing to collect more data to incorporate into your analysis before you proceed further.

Interpretation

After your data have been analyzed, you will turn to the interpretation phase of the data collection process. In this phase, you are looking at your dataset in order to try to figure out what the data are telling you.

In the interpretation phase, it is important to be aware of biases. In traditional statistics, bias is something that we try very hard to avoid. While it’s important to not let our own biases influence our interpretation of the data, the opinions, biases and perceptions of our stakeholders are an important and vital source of data that should be included and analyzed as a part of our data collection and should influence our interpretation of it. When infrastructure projects are built, for example, many people who are in favor of a project don’t want the project to disrupt their commute or create too much noise at their house. It’s important to hear and consider these concerns in the final design.

In order to generate buy-in from stakeholders in the process, it’s important to share interpretations of the data once they have been collected. These representations may take the form of summary reports, infographics, narrative stories, or through making data available in databases or electronic data structures. And while you may share raw data with your stakeholders, it’s more likely that they will be focused on what you’ve learned from the data and what you’ve identified as a resulting need. In design, these summaries would be tailored to the specific stakeholder group, and used to both validate your interpretation and to collect additional data.

Once themes have been identified, conduct a root cause analysis to ensure that you have fully identified “why” a problem or barrier may exist. Root cause analyses help us to understand whether the problem we’ve identified is actually the problem, or rather if it is a symptom of another problem. Consider this example of a restaurant: you go to a restaurant and order dinner. Everyone receives their orders, but it takes 45 minutes and all of the food is cold. Typically, one might say that the restaurant is “disorganized” or has “lousy service”. However, if you were in charge of reviewing the operations of the restaurant, you may determine that the waitstaff is inefficiently routed through the restaurant, or it is hard for the waitstaff to know when food is ready to be picked up, or the chef and the waitstaff are misinterpreting their notes and additional training is needed. Sometimes the root causes yield more obvious solutions (for example, the “I Voted” sticker is a simple solution that does get people to go vote).

A fishbone diagram can help with a Root Cause analysis.

Source: Wikipedia

Take a look at this entire process from start to finish in the healthcare field, using this example below.