Building Domain-Intelligent AI Systems

Building Domain-Intelligent AI Systems

When Generic Intelligence Meets Specialized Knowledge

Time to Complete: 30 minutes

PDF 5-Minute Warm-Up Activity can be downloaded above.

Who This Is For: This lesson is built for people who sit at the intersection of knowledge and machine -- and who feel the friction daily. That includes graduate students in information science, library science, computer science, knowledge management, digital humanities and organizational informatics who are grappling with how AI reshapes the disciplines they are entering. It is equally designed for working practitioners: research librarians and metadata specialists trying to evaluate whether AI cataloguing tools preserve or flatten subject expertise; knowledge managers and information architects in healthcare, law, finance, and scientific research who are being asked to deploy LLMs over proprietary corpora and need a critical framework for doing so; AI/ML engineers and product teams building domain-specific tools who have the technical capability but lack a vocabulary for articulating why their models keep failing domain experts; and ontology engineers and data architects who already understand knowledge structure but want rigor around where machine learning helps versus harms that structure. If you have ever been frustrated by an AI system that processes the right words but misses the meaning behind them -- or if you have been asked to trust an ‘AI-powered’ knowledge platform without clear criteria for evaluating it -- this lesson was written for you.

Real-World Application:

The epistemological tensions explored in this lesson are not theoretical -- they are active engineering and procurement problems. In clinical knowledge management, hospital systems deploying LLMs over medical literature face exactly the domain analysis challenge described here: a model trained on general text will conflate a cardiologist's use of ‘block’ with a neurologist's, producing retrieval failures with patient-safety consequences. In legal research platforms, firms evaluating tools like Harvey or Lexis+ AI must assess whether the system understands jurisdictional context or merely pattern-matches on statute text -- the same distinction Hjørland draws between processing terminology and comprehending epistemological frameworks. In enterprise search and RAG system design, knowledge engineers building retrieval-augmented generation pipelines must decide which taxonomies and ontologies to embed as retrieval constraints -- a direct application of the custom LLM design task in this lesson. In scientific publishing and research infrastructure, organizations like publishers, national libraries, and funding bodies are using automated classification to route manuscripts and allocate grants; without domain-sensitive design, algorithmic classification replicates and scales the interdisciplinary boundary failures this lesson teaches students to anticipate. Each of these contexts requires exactly the evaluation criteria -- technical performance alongside epistemological fidelity -- that this lesson asks students to develop.

Goal: You will develop advanced AI research skills by exploring how artificial intelligence can be designed to understand domain-specific knowledge, gaining practical experience with the epistemological tensions between machine learning's generalizing tendencies and the contextual depth required for specialized fields.

The Problem and Its Relevance

The rapid deployment of large language models has exposed a fundamental epistemological crisis: these systems excel at pattern recognition across massive datasets but struggle to respect the nuanced, context-dependent knowledge structures that define human expertise within specialized domains. When AI treats all knowledge as equivalent data points to be processed algorithmically, it risks flattening the rich epistemological frameworks that disciplines have developed over decades -- reducing a biologist's understanding of ‘species’ and a sociologist's interpretation of ‘community’ to mere statistical correlations rather than recognizing their incommensurable conceptual foundations. This creates a profound challenge for knowledge organization: How do you build AI systems that can navigate specialized domains without either oversimplifying their complexity or remaining trapped in isolated disciplinary silos? The tension between AI's drive toward universal models and the situated, socially constructed nature of domain knowledge threatens to reshape how we organize, access and validate information in ways that may undermine the very expertise these systems claim to support.

Why Does This Matter?

Understanding how AI integrates with domain-specific knowledge matters because:

(i) Epistemological foundations are shifting: AI introduces a machine-centric, data-driven approach to knowledge that challenges traditional human-mediated frameworks grounded in pragmatic realism and socio-cognitive understanding.

(ii) Context gets lost in translation: Current AI systems prioritize functional utility and pattern recognition over the deeper contextual sensitivity that domain analysis emphasizes -- potentially creating knowledge structures optimized for immediate retrieval but disconnected from disciplinary meaning.

(iii) Interdisciplinary integration remains superficial: While AI can connect disparate data sources across fields, this integration often amounts to data amalgamation rather than genuine epistemic synthesis that respects each domain's unique frameworks.

(iv) Gatekeeping and accountability are unclear: Traditional expert communities have validated and curated domain knowledge, but AI-automated classification raises questions about who controls information accuracy and how algorithmic decisions can be transparently verified.

(v) Dynamic versus static knowledge structures: AI enables real-time adaptation of classifications based on emerging patterns, moving away from stable, human-created taxonomies toward fluid systems that may or may not align with disciplinary conventions.

(vi) Human cognition cannot be fully simulated: Despite advances in natural language processing, AI lacks the intentionality, social motivation, and meaning-making processes central to how humans construct and interpret domain knowledge.

The challenge of integrating AI with domain analysis represents a collision point where computational efficiency meets epistemological depth, requiring careful consideration of what knowledge truly means when machines participate in its construction.

Three Critical Questions to Ask Yourself

• Do I understand the difference between AI that processes domain-specific terminology versus AI that genuinely comprehends the epistemological frameworks underlying specialized knowledge?

• Can I identify when AI-driven knowledge organization enhances domain analysis versus when it risks diluting contextual sensitivity through overgeneralization?

• Am I able to evaluate the trade-offs between computational efficiency, interdisciplinary connectivity, and preservation of domain-specific nuance when designing AI systems for specialized fields?

Roadmap

Examine this paper and identify how domain analysis (DA) traditionally functions as a socio-cognitive framework that emphasizes how culture, language and context shape knowledge within specific communities.

Working in groups, your task is to:

(i) Select a knowledge domain where AI integration would be valuable but potentially problematic -- this could involve scientific research (biology, physics), professional practice (medicine, law), cultural heritage (archival science, digital humanities) or interdisciplinary fields (environmental studies, information science).

Guidance: Choose a domain where terminology carries deep contextual meaning that extends beyond surface-level definitions.

(ii) Analyze the epistemological characteristics of your chosen domain. Describe how knowledge is produced, organized and validated within this community. Identify the specific taxonomies, conceptual frameworks and discourse practices that make this domain distinct. Explain what would be lost if AI treated this specialized knowledge as generic data.

(iii) Design a custom LLM approach that would enhance rather than undermine domain analysis by addressing:

• Which of Hjørland's eleven methodological approaches to knowledge organization would your AI system need to support (literature analysis, bibliometric studies, user studies, classification analysis, etc.)

• How you would ensure the system respects three critical dimensions:

• 1. Contextual sensitivity: What mechanisms would preserve domain-specific meaning?

  2. Social embeddedness: How would the system account for the communities and practices that shape knowledge?

  3. Epistemic diversity: What safeguards would prevent the flattening of disciplinary differences?

• At least 2-3 specific technical approaches from the paper (NLP for domain terminology, automated classification refinement, interdisciplinary mapping, bibliometric pattern analysis) that would support your design

(iv) Map the integration tensions inherent in your approach. Identify specific conflicts between AI capabilities and domain requirements: Could rapid algorithmic updates disrupt carefully constructed taxonomies? Could cross-domain pattern recognition obscure important epistemological boundaries? Would efficiency gains come at the cost of interpretive depth?

(v) Develop evaluation criteria that measure both technical performance and epistemological fidelity. Explain how you would assess whether your custom LLM genuinely enhances domain analysis or merely automates superficial tasks. Consider both quantitative metrics (accuracy, retrieval effectiveness) and qualitative dimensions (expert validation, contextual appropriateness).

(vi) Compare your approach with at least two alternative AI integration strategies: pure automation (where AI independently classifies and organizes), human-in-the-loop hybrid systems, or domain-agnostic general-purpose models. Create a structured comparison examining how each balances computational power with domain sensitivity, addresses interdisciplinary challenges, and maintains epistemological integrity.

Guidance: Focus on realistic implementations rather than ideal scenarios—acknowledge where technical limitations require compromise with theoretical principles.

Individual Reflection

Respond to your group's work by articulating what this exercise revealed about AI and specialized knowledge. Consider including:

• How this activity shifted your understanding of what constitutes ‘intelligence’ when comparing human domain expertise with machine pattern recognition

• Whether you gained insight into why domain experts sometimes resist AI tools that appear technically sophisticated but epistemologically shallow

• What you learned about the relationship between efficiency and depth in knowledge organization: when speed and scale enhance understanding versus when they sacrifice it

• How you might evaluate AI research tools differently after recognizing the distinction between data processing and genuine comprehension of domain contexts

• Whether this experience changed your perspective on what makes knowledge ‘organized’ versus merely ‘accessible’

Bottom Line

Custom LLMs succeed when they augment rather than replace the socio-cognitive processes through which domain communities construct meaning -- this requires explicit design choices that prioritize epistemological fidelity over computational convenience. The four categories of AI integration -- global automation, localized assistance, architectural adaptation, and interface mediation -- each sacrifice different aspects of either machine efficiency or human contextual understanding, with none offering a universal solution across all domains. Your objective is not to eliminate the tension between AI's generalizing power and domain specificity's contextual depth, nor to resist AI's transformative potential out of disciplinary protectionism. Rather, it is to develop the critical literacy needed to design systems that respect how knowledge functions within specialized communities while leveraging computational capabilities to address genuine research challenges. When you can articulate why certain epistemological commitments matter, which domain characteristics must be preserved, what forms of human expertise remain irreplaceable, and where algorithmic assistance genuinely helps rather than hinders, you have cultivated the AI research literacy essential for building systems that serve scholarship rather than merely processing it. This understanding proves valuable whether you are developing AI tools for academic research, evaluating vendor claims about ‘AI-powered’ knowledge platforms, or simply navigating a world where the question ‘Can machines truly understand specialized knowledge?’ has profound implications for how we organize, access, and trust information across increasingly AI-mediated environments.

#CustomLLMs #DomainKnowledge #EpistemologicalAI #ContextSensitiveComputing #HumanMachineSynergy

{"@context":"https://schema.org","@type":"LearningResource","name":"Building Domain-Intelligent AI Systems: When Generic Intelligence Meets Specialized Knowledge","description":"A graduate-level active learning lesson examining the epistemological tensions between large language models and domain-specific knowledge structures, with applied LLM design and evaluation tasks.","dateModified":"2026-03-06","version":"1.0"}, {"educationalLevel":"Graduate / Advanced Undergraduate","timeRequired":"PT30M","inLanguage":"en-US","url":"https://www.marvinuehara.com/ai-literacy-lesson-plans","isAccessibleForFree":true,"learningResourceType":"Lesson Plan"},

{"teaches":["domain analysis","knowledge organization","epistemological AI design","custom LLM development","domain-specific NLP","ontology engineering","taxonomy-aware AI","AI knowledge grounding","RAG pipeline design","knowledge graph integration","fine-tuning domain models","AI literacy","human-in-the-loop systems","AI bias in classification","information architecture for AI","interdisciplinary knowledge integration","AI evaluation frameworks","epistemic fidelity assessment"]}, {"keywords":["domain-intelligent AI","custom LLMs","knowledge organization","epistemological AI","domain analysis","AI for specialized fields","domain-specific NLP","RAG pipelines","AI knowledge systems","ontology-driven AI","information science AI","AI literacy","knowledge management AI","practitioner AI tools","AI for research","subject matter expert AI","domain expert augmentation","AI classification bias","taxonomy design","LLM fine-tuning for research"]}, {"audience":{"@type":"EducationalAudience","educationalRole":["Graduate Student","Research Librarian","Knowledge Manager","AI/ML Engineer","Data Architect","Ontology Engineer","Information Scientist","Digital Humanities Scholar"]},"author":{"@type":"Person","name":"Marvin Uehara","url":"https://www.marvinuehara.com"}}