Many organizations begin their AI adoption in a familiar way. Employees experiment with ChatGPT, Copilot, Gemini, Claude, or another platform. Different departments identify possible applications, and leadership authorizes a few pilots to see what the technology can accomplish.
These experiments can create valuable momentum. However, as organizations move forward, they will likely encounter more difficult questions:
- Which platforms should the organization support?
- How should different departments coordinate their pilots?
- When would a local large language model work better than a cloud service?
- Does the organization have enough computing capacity?
- How will leadership keep AI secure, governed, and connected to business outcomes as its use grows?
In short, an organization cannot build a long-term AI capability by approaching each of these decisions separately. Instead, the organization must construct an AI foundation, considering how its platforms, infrastructure, security, and governance practices will work together.
Select Platforms Around the Work
Unfortunately, there is no single AI platform that will work best for every department and application.
Microsoft 365 Copilot may fit naturally into an organization that relies heavily on Outlook, Teams, Word, Excel, PowerPoint, and SharePoint.
Gemini may offer similar advantages for organizations centered on Google Workspace and Google Cloud.
ChatGPT and Claude can support broader applications, including research, analysis, writing, software development, and custom workflows.
Meanwhile, SAP Business AI and Joule can bring AI into operational processes across finance, human resources, procurement, supply chain, and other SAP environments.
With these and other platforms available, leadership may feel tempted to select whichever one produces the most impressive response during a demonstration. However, the organization should first consider where its employees already work, where its data is stored, which permissions the platform must follow, and what the organization needs the AI to accomplish.
Guru’s comparison of business AI platforms recommends evaluating features, integrations, usability, implementation requirements, and total cost of ownership. The article also notes that subscription fees make up only one part of the investment. For instance, integration, training, administration, and change management can add considerable costs.
As a result, departments should not independently purchase several platforms with overlapping capabilities. An organization might instead establish one default productivity assistant, an SAP-centered operational layer, an approved development platform, and specialized tools for applications where they provide a measurable advantage.
This structure still gives departments room to experiment. At the same time, a central AI enablement team can coordinate licensing, security, integrations, training, and evaluation standards across the organization.
Each pilot should also begin with a process problem rather than a preferred product. For instance, they should determine what outcome the department is trying to improve, what information the platform will access, what will happen if the output is wrong, and how the organization should compare its results with the current process.
Determine Where Your AI Should Operate
Once the organization determines what AI systems will operate, leadership must decide where the AI will be hosted.
Enterprise cloud services may provide rapid implementation, advanced models, flexible capacity, and vendor-managed infrastructure. However, some organizations may need more direct control over sensitive information, model versions, network access, or offline availability. In these situations, a local LLM may become part of the solution.
When organizations discuss “building” a local LLM, they usually do not mean training an AI foundation model from the beginning. More often, they select an existing open-weight model, host it on company-controlled infrastructure, and connect it to approved organizational information.
To assist with the selection process, the GitHub guide on running LLMs locally compares tools such as llama.cpp, Ollama, Hugging Face Transformers, vLLM, and LM Studio. Some of these tools make early experimentation easier, while others support more customized or higher-volume deployments.
While a local LLM can give the organization more control, local does not automatically mean secure. For instance, the company must protect its model server, network, retrieval system, software dependencies, prompts, outputs, and organizational data. Additionally, it must manage updates, backups, access controls, monitoring, and incident response.
To help address these concerns, the SANS Institute’s risk-based AI security guidance identifies six important control categories: access control, data protection, secure deployment, inference security, continuous monitoring, and governance. The article also discusses risks such as model tampering, data poisoning, prompt injection, and regulatory gaps.
A hybrid approach between local and cloud hosting works best for many organizations. Sensitive and predictable workloads could run locally, while employees use an approved enterprise cloud platform for more general or demanding tasks.
Plan Infrastructure as a Business Portfolio
As the organization moves its pilots into production, infrastructure planning to solidify the AI foundation will likely become more complicated.
Where should leadership begin? According to AMD’s infrastructure-planning article, organizations should begin with their usage model rather than a hardware specification:
- How many employees and agents will use the system?
- Which models will they need?
- How much data must remain on-premises?
- What response times are required?
- Which databases and business applications will the agents access?
Additionally, agentic workflows may depend on CPUs for planning, tool execution, permission checks, database queries, retrieval, and coordination. Because of this, storage, memory, networking, software, power, cooling, and employee expertise can become equally important.
An IDC report sponsored by Intel argues that many organizations will eventually use “fit-for-purpose” hybrid infrastructure. Some workloads may remain in public clouds, some may operate in private environments, and others may run on existing general-purpose systems. The report also cautions organizations against assuming that every AI application requires specialized hardware. For example, smaller models, retrieval systems, data processing, and some inference workloads may operate effectively on existing infrastructure.
Before making a major purchase, an organization should test representative workloads and identify the actual bottleneck. Is the model too slow, or is the retrieval system delaying the answer? Are GPUs underused because storage or CPU resources cannot keep up? Would cloud capacity provide more flexibility while demand remains uncertain?
With this information, the organization can create several growth scenarios and revise them as actual usage emerges. The Introl capacity-planning article recommends monitoring utilization, adding capacity in stages, standardizing configurations, and using cloud resources to address temporary demand spikes.
Keep Governance Connected to Growth
Governance ties the AI foundation decisions together:
- Which AI systems will the organization use?
- Which information can the systems access
- Who owns the systems?
- How will leadership undergo evaluation?
- When should AI systems be changed or retired?
The NIST AI Risk Management Framework organizes this work around four functions: “Govern, Map, Measure, and Manage.”
Fortunately, an organization does not need to create a large new governance department before it can begin. Instead, it could start with a cross-functional group that represents business leadership, IT, security, legal, finance, human resources, and the affected operational teams.
From there, the group can maintain an inventory of AI use cases, classify risks, approve platforms, coordinate infrastructure, and compare investments with measurable outcomes.
AI adoption may begin with one pilot, but it will not remain one technology project. Over time, it will become a collection of platforms, data sources, integrations, models, employees, and infrastructure decisions. Organizations that prepare for this transition will find themselves in a stronger position to scale their AI foundation without losing control of their costs, information, or business objectives.
Over the next three articles, we will explore how to coordinate AI platforms and pilots, evaluate local LLMs and their security, and build the infrastructure and governance needed to support AI over the long term.
Leave A Comment