Why Consider Assumptions and External Factors?

Why Consider Assumptions and External Factors?
The Make or Break of Your Program

More than likely, you have taken a long road trip. In preparation for your road trip, you had a certain checklist of things that had to be considered before you could take off.  First of all, you had a specific set of expectations that 1) your car can make it (mechanically speaking), 2) your tires and their pressure are good, 3) you have sufficient money to cover gas, food, and lodging, 4) the route you have selected is the safest or quickest (whichever is most important to you), and 5) you have accurately calculated how long the trip will take which obviously affects the overall cost of the trip in time and money. But then, there are things outside your own control that you have to think about as they could ultimately make or break your trip. These environmental factors may be things like icy roads, blizzards, hurricanes, etc., depending on where you were traveling and at what time of the year. Other factors, such as road construction, car accidents, and alternate routes, may also unintentionally play into your trip.

Without contemplating all of these things, you may not actually make it to your destination. Yes, you could make it part way, but without sufficient consideration and contingency planning, you could get stranded or have to turn around.

In this scenario, would you ever consider planning a family vacation without contemplating the possibilities, especially having your littles on board?  Probably not.

In the same way, without contemplating all that you believe about how the project will work (assumptions), as well as what could potentially influence your project outside of your control (external factors), you are setting your program or project up for failure. To mitigate against potential problems, it is critical to think through each of these elements as you are planning and developing your program logic model[1].

When considering your assumptions, reflect on your principles, beliefs, and ideas about the:

• Problem/situation
• Resources/staff
• Way the program will operate
• Expected program achievements
• Knowledge base
• External/internal environment
• Participants (their behavior, motivations, how they learn, etc.)

A key question to help you process your assumptions is: Why do you believe the planned work will achieve the expected outcome(s)?

When considering your external or environmental factors (surrounding circumstances) reflect on the:

• Political environment
• Economic situation
• Social/cultural context
• Geographic/natural constraints

A key question to help you process your assumptions is: How can the [insert element above] affect our program and potentially influence our outcome(s)? Influences can be both positive and negative, but either way, they must be considered and accounted for.

Programs, like life, do not happen in a vacuum.  It is often the lack of accurate or overlooked assumptions and external factors that are the basis for program failure or poor results.

Just as you would never consider taking your family on a trip without proper preparation, take the time to work through these elements and develop contingencies for your programs should any of your present factors change. It’s the only way to ensure your program has the greatest chances for success.

[1] Please see our White Paper about Logic Models at http://eliteresearch.com/whitepapers  if you need more information.

How Do We Get Buy-In From Our Organization?

How Do We Get Buy-In From Our Organization?

Having a data-informed culture is all about better decision-making. It’s using data to make informed decisions.  In the words of H. O. Maycotte, “being data-informed is about striking a balance in which your expertise and understanding of information plays as great a role in your decisions as the information itself.” And it is in this place of balance that organizations can capitalize on huge opportunities.  The MIT Sloan Management Review found that top-performing companies are three times more likely than lower performers to be sophisticated users of data – meaning those that know how and when to appropriately use data in their decision-making are more likely to outperform their counterparts. It also found that organizational management and culture rather than with data (itself) and technology are the primary barriers to effective use of data. If the obstacle to moving forward is with the organization itself, then the question really should be: How do we get buy-in from our organization so that we can begin to be sophisticated users of data?

It starts from the top. Leadership is critical to creating an organizational shift in culture. If your Executive Director does not understand the value of data and a data-informed mindset, this is where you must start. Educate through examples, case studies of similar mission and structure, etc. Executive leadership must see the value and begin to lead by example. Provide him or her talking points, statistics and numbers that help to advocate for change, demonstrations of analytics, and if possible, create a vision for what you are trying to do and the questions you are trying to answer (addressing your major obstacles or growth opportunities).  

A vision map should be segmented into small enough pieces where you can begin to execute in a data-informed way. These segmented steps may include a series of small measurement pilots; by breaking the vision down into smaller pieces you are setting your organization up for small wins.

Do not underestimate the power of small wins - they should be celebrated in your organization. It may seem like you have only created a small ripple, but a series of small ripples (i.e., wins) can have a tidal wave effect later on. 

Also, never underestimate the power of transparency – to your donors as well as your staff.  As data begins to surface, there will be things you celebrate for having achieved a step in the right direction. But, data will inevitably tell you where you are not doing so well, and that too, should be celebrated. Knowing where you need to focus your attention (instead of playing a game of Marco Polo with your resources and constituents) allows you to pause, recalibrate, and get back on course to effective work.  If we are honest, your staff probably already has a decent handle on what is or isn’t working since they are the ones on the proverbial “front lines;” your ability to validate data through their intuition and experience will build a more unified team. Yes, some may not see the value in data given their actual experience, but in this data age we live in, data will speak more loudly than any internal report.

With donor profiles changing and data being the new currency in town, nonprofits must be able to demonstrate the strategic use of data in their decision-making. Data does not have to trump experience; it must complement it. Federal and state agencies, large foundations, and the emerging new donor all require data for all funding. Organizations that can embed a data mindset (and appreciation for it) will be the ones that continue to thrive.

Remember that cultures never shift overnight.  It will take time. Realistically, it may be at least a five year process, so give your nonprofit grace along the way.

What Are Some Data Collection Challenges and How Do You Overcome Them? (Part 3)

What Are Some Data Collection Challenges and How Do You Overcome Them? (Part 3)

Getting it right or as close to right as possible is critical when collecting data.  Failure to properly collect data include the inability to answer your research questions, inability to repeat and validate the study, distorted findings, wasted resources, misleading recommendations and decisions, and harm to participants.  

There are many things that could go wrong when collecting data, which may ultimately compromise the trustworthiness of your findings.  To help you think through potential challenges and how to address them, here are some challenges and advice to consider as a nonprofit organization.  

This article focuses on the challenges presented After Data Collection.

• Challenge: Disorganized databases.

Consideration: Take every effort to create clean and organized databases before the data entry has begun (do not wait until you need the database).  Provide step-by-step instructions and have trial runs with those entering the data.  To minimize data errors, include drop-down options.  In addition to actual survey data, be sure to include: survey respondent’s ID, data enterer’s ID, and date of entry. 

• Challenge: Lost or overwhelming amount of data.

Consideration: To stay on top of incoming data, hire or assign a separate data entry team who daily enter data.  Each person should be assigned specific survey number ranges to complete daily.  The person who entered data should be noted in the database. 

• Challenge: Invalid data entry.

Consideration: When creating the database, be sure to “lock” fields by providing drop-down options for data entry selections.  For every person working on data entry on a given day, randomly check 10% of data being entered.  If any part is entered incorrectly, STOP and recheck all the data that was entered that day.  

• Challenge: Duplicate or missing data. Inevitably, it happens with every data set.  If duplicate or missing data is left unattended, it will skew your results. 

Consideration: Learn how to identify and remedy these issues within the database/analysis programs you are working with. 

Take every effort to set up appropriate measures before data collection begins (quality assurance) and then to follow the protocols during and after data collection (quality control). 

Be diligent in the process.  It is the only way you will be able to trust your findings.

How Can You Reduce Costs for Research & Evaluation

How Can You Reduce Costs for Research & Evaluation?

It’s true, beginning and maintaining a research and evaluation (R&E) program costs money.  Additional budget lines can be a headache to advocate for with your board, but they are definitely worth the effort because of the benefits they provide your nonprofit.  Having the ability to demonstrate your nonprofit’s value to the community and obtain additional funding for needed programs far outweigh the effort.

Still, oftentimes you are faced with the issue of cutting other budget lines to make room for R&E items that don’t always feel necessary. 

As a vendor, it is a quite common to hear that R&E budgets are not priority, they are the cuts necessary to make budget, and ultimately these costs preclude an organization from conducting any evaluative ventures. 

This does not have to be the case.  There are many ways that nonprofit organizations can run successful R&E programs without breaking the bank.  The key is using funds strategically and embracing the idea of organizational learning.

Knowing how and where to use your R&E funds is critical.  Too often, nonprofits spend money on “research” that does not tell them much.  Simply conducting a survey or talking to people does not mean you’ve done it right.  Here are some tips that may help you in this area:

• Be sure you are asking the right questions. This is actually a lot harder than it seems.  It’s good to consider brainstorming with people who know whether or not the information you want is attainable and how best to go about obtaining it.
• Determine what you will do with your findings once you get them and commit to using them.
• Consult with an expert on the methodology and sample size so that you may appropriately generalize your findings.
• Spend the time to design and properly validate a survey (may need to include qualitative research elements to do so). If working across languages, comprehension (pilot) testing must be considered as well.
• Reflect on alternate data collection approaches. There are many options available – train the trainer, train locally, online, mobile, etc. Each have unique advantages (and disadvantages).  Find others who have had experience in these areas and talk to them, especially if you work internationally.
• Consider qualitative research in lieu of survey or quantitative research to explain the why’s and how’s (depending on your research questions). It does take a little longer, but it does not require the same amount of approvals, and it yields information-rich data.
• Collaborate with others[1] of similar vision and mission. By pooling resources and questions, you are able to find much more than you could if you were flying solo.

The other aspect of saving money with R&E is investing in your staff. Organizational learning is defined as the process of creating, retaining, and transferring knowledge within an organization and thus improving over time as it gains experience. Organizational learning is your nonprofit’s coupon of sorts. Yes, you have to spend money upfront to train and do research, but you are saving money by being able to do much of it yourself later. Here are some tips that may help you in this area:

• Utilize internships to identify secondary research concerning topics of interest to your organization; make this information available to the entire group.
• Identify a person responsible for overseeing R&E. This central person should manage (not necessarily perform) the rolling out of projects.  This person should also explain how processes are conducted.
• Research objectives, methodologies, and findings should be shared across the organization – regardless of whether findings are “good” or not.
• Low performance should be celebrated as much as high performance because you have now identified areas of growth and opportunity.
• Leadership should continue to corporately champion the vision and reasons for R&E, celebrating the little wins along the way.
• Cross pollinate ideas between departments. Sometimes, there are untapped innovative ideas based on the experience of another team.  Share your research findings and ask for feedback.  Learn from each other.

Because changing donor profiles and evaluative data (now often a requirement for funding consideration) continue to trend upward, take the time to figure out how you can be strategic with your R&E.  Do not take the easier road and have an outsider do the work for you; get the experience doing it yourself.  From this experience, your nonprofit is able to create knowledge, and it is from this knowledge that your organization grows.

[1] See article How Can Collaborations Help Us in Research & Evaluation at http://eliteresearch.com/nonprofit-articles