The philosopher and psychologist John Dewey wrote "...[A] problem well put is half-solved."
"... The way in which the problem is conceived decides what specific suggestions are entertained and which are dismissed; what data are selected and which rejected; it is the criterion for relevancy and irrelevancy of hypotheses and conceptual structures."(1)
Long before I had read Dewey, my training as an intelligence analyst had taught me the importance and implications of framing a question. When I receive a question or problem from a client, I make sure I understand the intended outcome and not just the words spoken. This is because a client's intent can sometimes not be realized by what they are requesting or there might be a better approach --- simpler, faster, multi-purpose, etc. If I had failed to understand the intent, I might have provided what is asked for while not providing what is needed.
Described below are four examples of where I creatively solved a problem by redefining the immediate problem into a new problem statement.
How many traffic lights in the United States?
Global Traffic Technologies (Oakdale, MN)
The problem
My client, Global Traffic Technologies, wanted to know the size of the market for their traffic control products and services. In addition, they wanted to know the demand for every US city with a population over 5000 people.
The problem about the problem
I discovered that cities usually cannot tell you how many traffic lights they have. Some will even say that information is confidential. In any event, there are almost 30,000 municipalities in the 50 US states and the District of Columbia. Collecting this information would be impractical even if it were available.
Redefining the problem
In mulling the problem, I thought about how traffic density is the primary reason for a traffic light and how the number of road intersections where that density exists largely determines the number of traffic lights. Unfortunately, counting intersections in 30,000 US municipalities, even without considering their traffic density, was as problematic as counting traffic lights. Were there easily obtained stand-in metrics for traffic density and number of intersections?
I decided that population density could stand-in for traffic density and total dry land area could stand in for the number of intersections. Of course, not everyone sitting at a traffic light during their daily commute would live in the traffic light's town, but "directionally", population density seemed a good stand-in for traffic density. I later confirmed this when validating my solution.
The solution
Using population density and dry land area data from the US Census Bureau and other sources, I worked with the client to define an algorithm that used these metrics to make an educated guess about the number of traffic lights in any given US city. In general:
- The population density determined if a city likely needed traffic lights at all.
- The dry land area (which excluded bodies of water) determined how many traffic lights they might have or need given the population density.
This algorithm was calibrated using the actual traffic light counts that were available for a few cities of various sizes.
I then began indentifying every municipality in the 50 U.S. states plus the District of Columbia. In the process, I discovered the U.S. had approximately 30,000 municipalities. These are political entities below the level of state. I recorded each municipality's name, state, and designation (hamlet, town, city, county, etc.) along with its population density and dry land area. I obtained this information using US Census Bureau and other sources. I combined this information with CRM data and made it available to my client in a self-serve reporting tool you can learn about here.
Why finding 30,000 candidates was easier than finding three
Wells Fargo (Minneapolis, MN)
The problem
Wells Fargo Home Mortgage (WFHM) was conducting a major recruiting effort to increase their mortgage loan officer staff throughout the country. They were doing this by recruiting the best people from the competition. For this project they assembled a staff of 12 recruiters. They also contracted me to identify candidates. The criteria were usually very specific and involved foreign language fluency, specialization in specific loan types, and geographic proximity to the office for which WFHM was recruiting. My client always wanted the best performing loan officers they could get and they always wanted at least three candidates to consider for each vacancy.
The problem about the problem
I soon realized that the time needed for me to identify three good candidates for just one vacancy meant I could not maintain a steady stream of good candidates to all 12 recruiters.
Redefining the problem
Since I was already using competitor's websites to search for candidates, I wondered how I might leverage those websites for a more efficient approach. I had been aware of webscraping, but had never tried it. Could I webscrape the websites of every mortgage loan provider in the United States and extract details about all their mortgage loan officers? If I could, I could then put the results into a database that the recruiting staff could search themselves. Research and a pilot project proved I could do this.
The solution
With my client's permission, I changed the nature of my work from finding candidates for individual vacancies to the audacious project of capturing, in theory, every mortgage loan officer in the United States along with details about their qualifications.
Using specialized software, I created webscraping templates for the websites of dozens of mortgage loan providers. I purchased a database of US Zip codes and fed it into my automated webscraping. Every night as I slept, my software would use the website's loan officer search feature to retrieve all such officers by Zip code. Along with their name and contact information, my templates extracted the details of their skills and sales awards. The results were saved in an MS Access database. In the morning, I would clean up the results and append them to my growing database on the company network.
Along with my database, I created a user-friendly interface for recruiters to run their own searches. You can see screenshots of this interface here.
After the recruiting initiative was completed, Wells Fargo continued to use my database for routine recruiting. I was contracted several times over the next few years to refresh the data. At its largest, I had captured over 30,000 active mortgage loan officers.
How to validate customer addresses for free?
Global Traffic Technologies (Oakdale, MN)
The problem
After each trade show in which my client, Global Traffic Technologies, would participate, they would have 100+ names and addresses of potential government customers for their traffic control devices. At first recorded in handwriting, then typed into a spreadsheet, the results needed to be verified, corrected where needed, and deduped before being uploaded to my client's NetSuite CRM database.
The problem about the problem
For-fee services could be used for validation, as could the US Postal Services lookup API. However, for-fee services would be too expensive for small, infrequent counts. The USPS API is limited by its terms- of-use, making it impractical.
Redefining the problem
Being familiar with the online tools of the US Census Bureau, I knew they had a tool that would let me upload a file of addresses to get the census bloc designation appended to each. This was possible because the Census Bureau maintains a database of every US address so it can conduct its surveys of people and businesses. If I uploaded a list of business addresses with names removed, could I leverage this free tool to validate those addresses?
The solution
I tested the US Census Bureau's bulk upload tool with a sample file containing several hundred addresses both correct and intentionally incorrect but based on real addresses. Although the Census Bureau tool reported an occasional false negative ("no match") when the address was actually valid according to other sources, the tool was largely correct.
Encouraged by this, I uploaded an Excel file of ~100 recently acquired tradeshow addresses (sanitized of both people and company names). Within two minutes, I received my Excel file back with valid addresses confirmed as such. There was only a small percentage that I then needed to research before adding the contacts to my client's CRM database. You can see a short presentation on how I used this tool here.
What to do next (in investigations of international heroin smuggling organizations)?
Drug Enforcement Administration | Northern New Jersey HIDTA Task Force (Newark, NJ)
The Northern New Jersey High Intensity Drug Trafficking Area Task Force (HIDTA) was a joint federal, state, and local drug enforcement task force. It investigated and eliminated international heroin smuggling organizations operating through Newark’s airport and seaport. I was contracted to train and oversee a staff of intelligence analysts as well as to provide intelligence support myself.
The problem
The Task Force always had six to eight major investigations going on at any time and several in the early stages. The analysts I would supervise were all military reservists, most on six-month temporary assignments to our task force. While most of them were trained intelligence analysts, drug smuggling investigations would always be something new for all of them. Because of the urgency of the Task Force’s work and the limited time an analyst would be available to the Task Force, I had to find a way to get them trained and productive quickly. This training would have to be repeated every six months as the reservists rotated in and out. I summarized my situation as follows:
- How can my analysts be largely self-directed in both their training and in their support of an investigation?
- How could my analysts speak with confidence when advising their investigator on what next steps should be?
The problem about the problem
Investigations were inherently complex for both the investigator and the intelligence analyst supporting them. That was the problem about my problem. However, I knew from my military intelligence training that the approach to complex problems was to break them down into smaller, simpler ones. The tricky part would be to organize all the small tasks in such a way that I did not simply restore the complexity from which the tasks came. I also had to organize the tasks into a progression so that the next step was always obvious.
My solution would have three parts.
Solution part 1: Recording the sequence of investigator actions and what questions had to be answered to enable each action
Working with the investigators, I identified the usual order of their actions in an investigation. I say "usual order" because investigations often required one or more deviations from a typical sequence. These actions included things like identifying targets for surveillance, for phone bill subpoenas, and for pen register warrants.
Next, working backward from each investigator action, I recorded the questions whose answers would enable the investigator to take a particular action. I had identified these questions earlier during my requirements gathering sessions with the investigators.
Solution part 2: Recording how to answer the questions
For each set of questions recorded in my list, I created a short procedural manual on how to find their answers, what resources were needed, and what the quality checks and format for the results were. These were the procedures my soon-to-arrive intelligence analysts would need to perform. I gave each procedure a unique ID number which I recorded in my list next to the applicable questions to be answered.
Solution part 3: The HIDTA Intelligence Cycle chart
Bringing all this together was a large wall chart I created called “The HIDTA Intelligence Cycle.” My use of a circle to depict this process was to mirror that of the intelligence cycle in military intelligence doctrine. Also, without an end point, my cycle would remind the analysts that an investigation was not finished simply because the last task had been completed.
My wall chart consisted of three concentric rings that incorporated my list of investigator actions, their corresponding questions, and the IDs of the procedures for answering those questions.
Inner ring: The specific task of the intelligence analyst. Each task was identified by the ID of its corresponding procedure manual, so the chart and the documentation were synchronized. The use of an ID rather than the title of the manual ensured the analyst focused on the investigator action and the questions whose answers would support that action.
Outer ring: The subsequent activity of the investigator. This could be to organize a surveillance plan, subpoena particular types of documents, or seek telephone monitoring.
Middle ring: The bridge between the analyst's task and investigator’s action. These were the questions to be answered by the analyst task. Those answers would enable the investigator to do the action necessary at that particular stage of the investigation.
Years later, I realized my chart was a series of user stories: "As an intelligence analyst, I need to do X so the investigator can do Y."
By looking at the chart, an intelligence analyst knew what questions they had to answer next, what the purpose was, and where to find my instructions on how to answer them. They also saw how that intelligence support moved the investigation forward. When investigators returned to the office and wanted an update from their analyst, the chart provided a common reference in discussions of progress, obstacles, and next steps. Because the likely next step was depicted on the chart, the new intelligence analyst could speak with confidence with their investigator about what they might accomplish next.
The depiction above is an abstract of this chart which, due to its confidentiality (not to mention my fading memory), I cannot represent in detail.
The Northern New Jersey HIDTA Task Force was later merged with the New York City HIDTA to form the New York / New Jersey HIDTA.
- John Dewey, "Logic: The Theory of Inquiry", John Dewey: The Later Works, 1925-1953
(Carbondale, IL: Southern Illinois University Press, 2008), p. 112.
Copyright Will Beauchemin 2024