AI Trailblazers: Joe Manfredonia On How Flatiron Health is Leveraging AI for Cancer Research and Employee Productivity

In this interview of the AI Trailblazers series, we sat down with Joe Manfredonia, Director of Analytics & Practical AI at Flatiron Health. Flatiron is an industry-leading health tech company dedicated to transforming cancer care through technology and data. From utilizing AI to extract insights from unstructured medical data to adopting tools like GitHub Copilot and Dashworks to unlock employee productivity, Flatiron's holistic approach offers valuable lessons for organizations looking to adopt AI responsibly.

Q: Can you tell us a bit about Flatiron Health and how AI fits into your company's mission?

Joe: Flatiron is a health tech company dedicated to improving cancer care and advancing research using real-world data. Our mission has been and continues to be to improve and extend lives by learning from the experience of every person with cancer. We offer point-of-care oncology-specific EHR software used by about 40% of community oncology practices in the United States, which helps care teams deliver care to their patients.

Through that, we're collecting really rich data. In fact, we have over 5 million patient records available for research in our real-world databases. Clinicians, researchers, and regulators all rely on Flatiron technology and that data to learn faster and power smarter care for every patient.

One of the core ways we've been leveraging both traditional machine learning approaches and now AI is unlocking insights from unstructured data. We use these technologies to extract valuable information from doctors' notes, visit transcriptions, radiology and pathology reports, imaging scans and more?. There's a lot of rich information locked inside these unstructured sources that we previously relied on human abstraction teams to extract. Now, we're leveraging traditional ML and AI approaches to do that same work at a significantly greater scale, which is really exciting.

Q: Beyond your core product offerings, how are you implementing AI for your employees?

Joe: In addition to building AI into our core product and service offerings, we're also focused on ensuring our employees have access to the latest AI tools they might be familiar with in their personal lives. Every minute we help an employee be more productive or find information more quickly is another minute we can give back to furthering our mission of improving and extending the lives of cancer patients.

One of our core challenges, even before large language models and AI were making headlines, was dealing with diverse information sources and documentation. As Flatiron grew, different business units naturally stored their documentation in different ways and places. Increasingly, we've been taking more of a pan-Flatiron approach to solving problems, which creates opportunities for synergies across those boundaries.

It's been challenging for our employees to figure out where to find certain pieces of information, who knows what, or who to ask questions. We've managed well enough over the years, but with the rise of AI, we saw an opportunity to address this challenge more effectively. That's where Dashworks came in—rather than building our own capability, we could take a partnership approach that gives us access to the latest AI models while connecting them to Flatiron-specific context.

Q: What teams or departments are finding the most value from AI-powered search and retrieval of internal data?

Joe: There's certainly value in the general search and retrieval use case across the company. Teams are finding tremendous value in quickly locating information through a natural language chat interface and having key points synthesized in a digestible, readable, and concise way.

Flatiron’s core principle has always been to combine AI and ML tools with expert wisdom and use both to their best ability and there are some specific areas where these AI tools are  adding tremendous value for our teams in very focused ways. For example, we recently reached our 1,000-publication milestone, which highlights the scale of our research efforts. Teams across Flatiron have found unique ways to leverage AI to find efficiencies across various stages of content development, helping to achieve this dramatic scale we’re working towards. 

Our customer support teams—both those supporting external clients and internal groups like IT—are also seeing great benefits. IT support handles everything from password resets to system access requests, and many of these are common, routine issues. These teams are finding value in using large language models with access to resources like our Zendesk ticket history to quickly generate responses to customer inquiries, which they can then fine-tune before sending.

Q: What AI models have you found work best for different use cases at Flatiron?

Joe: This landscape is rapidly evolving, and we're following industry developments just as closely as everyone else. Recently, our teams have been excited about Claude 3.7 as a general-purpose model—there's been a noticeable improvement in the quality of content compared to previously available models.

For coding specifically, the reasoning models are particularly valuable to our developer community, such as Claude 3.7 with its thinking capabilities. For general usage like search and retrieval, GPT-4o remains a quick and easy way to get started.

Different tasks seem better suited to different models: academic manuscripts and scaffolding benefit from Claude 3.7 Thinking, as does drafting longer form internal documentation like Confluence articles or blog posts. The reasoning models, especially those fine-tuned for STEM tasks, have been particularly valuable to technical teams in engineering who are doing development work or are looking to get up to speed on understanding a new codebase. We're developing a sense of which models to recommend for specific types of tasks, while recognizing that this landscape is evolving rapidly.

Q: How are you approaching AI education at Flatiron?

Joe: We've invested heavily in education around prompt engineering. When we conducted an AI literacy assessment nearly a year ago, a significant number of respondents indicated they had either never used ChatGPT or only used it once. It would be naive to think employees would immediately know how to use AI tools effectively without guidance.

Part of our rollout wasn't just "here's Dashworks, here's how to log in," but also general AI best  practices. We've developed content specific to common Flatiron workflows: if you're drafting a project charter, requirements document, or academic manuscript outline, here are specific prompts you can copy, paste, and customize to get started quickly.

We've also established guidelines for responsible AI use, including instructions to review output before sharing externally and include appropriate disclaimers, especially for academic work. While we believe our employees understand these principles, it's important to make these guidelines explicit.

Q: What does the rest of your AI tech stack look like?

Joe: We're customers of pretty much all the major cloud providers for our production workflows. Internally, alongside Dashworks, we offer GitHub Copilot for coding assistance, and we're experimenting with alternatives like Aider (an open-source, terminal-based assistant) and others like Cursor that have received positive attention in the industry.

We have a process for approving new tools while still encouraging experimentation. We're also recognizing that AI feature sets are being launched by tools we already use—Salesforce Einstein, Zendesk AI, Snowflake, Looker, Miro, and so on. We've expanded our focus to include not just standalone AI tools but also how to effectively engage with AI functionality in existing day-to-day tools.

Q: Have you noticed any concrete business outcomes or ROI from these AI tools?

Joe: Where we have data, we use it to get excited about the rest of our AI initiatives. For coding assistants, where we've done structured analysis, our findings suggest productivity improvements between 40-70% depending on the metric and time period, which aligns with industry research.

For general-purpose AI tools like Dashworks, while we haven't done structured analysis on productivity gains, the adoption metrics are compelling. Just a couple of months after rolling it out company-wide, we've reached over 50% adoption by monthly active users, and employees have asked over 50,000 questions so far in total.

Even with conservative estimates of time saved—say 3-5 minutes per question—that adds up quickly. These adoption and engagement metrics suggest employees are finding significant value in the tool, which is exciting to see.

Q: What obstacles have you encountered in AI adoption, and how have you addressed them?

Joe: In healthcare especially, there are significant concerns around privacy, security, and confidentiality. We did our due diligence there, and one helpful aspect of Dashworks' implementation is that we're not persisting our data somewhere new—Dashworks authenticates as a user to access information that user already has available, at prompt time. This means the data stays up-to-date and in its original location, and we're not using it to train models, which has been highly comforting from a compliance perspective.

With any new tool, there's always an adoption hurdle—asking users to change behavior and go to a new place for information. We addressed this with a comprehensive education plan, giving teams multiple notifications about the upcoming rollout, conducting a pilot with a smaller group first, and launching with a webinar to demonstrate usage. We've also maintained an engaged community on Slack where we highlight cool use cases and respond to questions, creating ongoing momentum rather than just launching and moving on.

Q: Looking forward, what AI capabilities are you most excited about?

Joe: I'm really excited about agents and agentic frameworks. Something I'm observing is that teams are getting creative with how they stitch together different tools to create workflows that, if you squint, start to look like agentic systems.

For example, we had a team using Dashworks' bot functionality to answer specific questions in Slack, and then they set up an Airtable hook listening to that Slack channel for messages matching specific patterns. Without supported integrations, teams are hacking together cross-tool workflows, which shows incredible creativity.

I look forward to a day when there are tools available for teams to create workflows where Dashworks might be just one node in the chain, or where they can chain together multiple prompts and share those templates. This will really accelerate innovation and impact with AI, not just at Flatiron but in general.

Right now, we have champions who are in the foreground experimenting and sharing their learnings, but there are many more users who will wait until there are playbooks and specific tools before jumping in. I'm looking forward to that tipping point where these agentic workflows become more accessible to teams that haven't had the energy or time to experiment.

Key Takeaways

• AI amplifies mission impact: Flatiron is leveraging AI not just for productivity gains, but to further their core mission of improving cancer care through better data analysis and research support.
• Balance human expertise with AI efficiency: Flatiron maintains a "human in the loop" approach, using AI to accelerate work to 30-60% completion before human review and refinement.
• Education is essential: Their successful AI adoption includes comprehensive education on prompt engineering and use case examples tailored to specific workflows.
• Model selection matters: Different AI models excel at different tasks - Claude 3.7 for academic writing and documentation, reasoning models for coding, and GPT-4o for general search and retrieval.
• Responsible use in healthcare: Clear guidelines prohibit the use of AI with patient-level information, with additional protocols for review and disclaimers on AI-assisted content.
• Impressive adoption metrics: With 50% of employees actively using Dashworks and over 50,000 questions asked, the engagement shows significant value even without formal ROI analysis.
• Looking forward to agentic systems: The future lies in connecting AI tools into workflows or "agents" that can accomplish more complex, multi-step tasks while remaining accessible to all employees.