TL;DR

• Enterprises are adopting Large Language Model (LLM) solutions to enhance productivity and knowledge management.
• Retrieval-Augmented Generation (RAG) systems are popular for improving LLM response quality by connecting them to knowledge bases.
• RAG systems must meet strict security requirements to ensure trust and compliance within organizations.
• This document explores security challenges in implementing RAG systems with LLMs and how IllumiDesk addresses these needs.
• Users can enhance LLM outputs by fine-tuning models with proprietary data or using RAG systems for contextual data retrieval.
• RAG systems offer cost-effective, scalable, and customizable solutions compared to fine-tuning foundational models.
• Key security challenges include securing data ingestion, access, and retrieval processes in RAG systems.
• IllumiDesk provides enterprise-grade security through robust access controls, integration with identity management systems, and secure data handling practices.

Security remains a top concern as enterprises embrace Large Language Model (LLM) solutions to streamline knowledge management and enhance productivity. Retrieval-Augmented Generation (RAG) systems, which connect LLMs to knowledge bases, are currently the most popular choice to augment the LLM’s context to improve the quality of the responses.

However, RAG systems must meet stringent enterprise security requirements to gain trust from senior technical teams and keep information sharing in line with how the organization operates and controls access to information today. This document describes the security challenges when implementing RAG Systems with LLMs and how IllumiDesk helps enterprises meet their security and compliance requirements.

Why Use RAG Systems with LLMs?

To improve the quality of the LLM outputs, users can:

1. Fine tune foundational models (FMs) with proprietary data: the process of fine tuning a model helps improve the quality of the model’s responsesreponses since the data is “baked” into the model. This process, however is expensive since it requires data preparation and compute time. There are also some inherent risks to fine tuning a model since the company may not want their data built into the model itself, once added it’s difficult or impossible to remove the data from the model.
   
   
2. Retrieve contextual data from RAG systems: RAG is widely used in LLM applications due to its benefits including: overcoming such as it’s ability to overcome LLM limitations like limited context window, expanding knowledge, and improvinge contextual relevance. For a RAG, users select the specific data sources they want the AI to leverage. RAGs also provides transparency, reduces hallucination, enables continuous learning, and offers flexibility, customizability, and scalability.

They key point here is to measure the cost/benefit of fine tuning foundational models vs using RAG systems, or implementing a hybrid approach (RAG + Fine Tuning) to improve the quality of the LLMs outputs. In most cases a RAG system is much cheaper to set up and maintain that fine tuning a foundational model.

Data Ingestion and Retrieval for RAGs - A Primer

There are two main touch points when using RAG Systems:

• Data ingestion: this step involves selecting the data sources and file types when ingesting data into the RAG System. Essentially, all data from all data sources are split into chunks of text, scored with an embed model, and stored in the RAG within a “vector space”.  All of these chunks of text are now combined into one integrated database that is agnostic to the source.
  
  
• Data retrieval: data is retrieved from the RAG System to augment the quality of the LLM’s responses. The LLM selects the most relevant chunks of text from the RAG and leverages them in the response back to the user.
  
  Both the data ingestion and retrieval flows introduce new security challenges for enterprises, as depicted below:

Figure 1: Storing and retrieving context from a RAG systems

Key Security Challenges for RAG Systems

The process of ingesting and retrieving data to and from RAG systems introduces several security challenges, such as:

1. How to secure the data ingestion pipelines
2. Secure access to the RAG system
3. Secure the information once in the RAG system
4. How to enforce end-to-end security when retrieving data from the RAG and how it’s shared with the generative AI model(s).

Enterprise-Grade Security Considerations

Enterprises should have the option to leverage existing security access controls when implementing, deploying, and maintaining RAG Systems with their LLMs.

The main ones are:

• Granular Access Controls: Enterprises need assurance that their existing Role-Based
• Access Control (RBAC) frameworks extend seamlessly to RAG systems: Users managed with systems like Microsoft Active Directory must maintain their defined permissions when accessing sensitive data in Confluence, SharePoint, and other repositories.
• Integration with Identity Management Systems (IdMS): Securely integrating RAG systems with enterprise IdMS, such as Microsoft Active Directory, ensures consistent user authentication and authorization across all data sources.
• Risk Mitigation in Data Storage: Sensitive enterprise data stored in RAG vector databases must remain secure and accessible only to authorized users.

Access Controls

Access controls are fundamental to applications handling sensitive information, ensuring data is only accessible to authorized users. This is particularly crucial in RAG (Retrieval-Augmented Generation) systems, where different users may have varying levels of access to indexed documents.

While most vector databases offer metadata-based access control, a more architecturally sound approach is to handle authorization separately from the database. This aligns with the principle of separation of concerns and enables more flexible post-query filtering.

The two key components for securing access to any data store are:

• Authentication verifies user identity through credentials like usernames and passwords
• Authorization determines what resources an authenticated user can access

Common Authorization Models

• Access Control Lists (ACL): Direct mapping between users and resource permissions
• Role-Based Access Control (RBAC): Permissions granted based on user roles
• Attribute-Based Access Control (ABAC): Access determined by user, resource, action, and context attributes
• Relationship-Based Access Control (ReBAC): Permissions based on connections between users and resources

Access Control with RAG Systems

RAG systems handle access control in two phases:

1. Ingestion Phase: Link system identities with their permitted resources
2. Query Phase: Filter results based on user permissions before returning responses

ReBAC is particularly well-suited for RAG applications because it focuses on resource relationships rather than endpoint management. This makes it ideal for post-query filtering, where the system must determine which documents a user can access based on their relationships to document categories.

IllumiDesk’s Approach to RAG Security

IllumiDesk incorporates best practices to ensure that RAG systems meet enterprise-grade security requirements. Here’s how our solution addresses the critical needs of your technical team:

Enforcing Robust Access Controls in RAG Systems

Access controls are fundamental to applications handling sensitive information, ensuring data is only accessible to authorized users. This is particularly crucial in RAG (Retrieval-Augmented Generation) systems, where different users may have varying levels of access to indexed documents.

While most vector databases offer metadata-based access control, a more architecturally sound approach is to handle authorization separately from the database. This aligns with the principle of separation of concerns and enables more flexible post-query filtering.

Seamless Integration with Identity ManagementManagment Systems (IdMS)

• IllumiDesk integrates directly with your existing IdMS, such as Microsoft Activity Directory, Okta/Auth0, et al to maintain consistent access controls:
• Single Sign-On (SSO) Integration: Users authenticate through Microsoft Active Directory, ensuring a seamless and secure login experience.
• Permissions Synchronization: IllumiDesk synchronizes access permissions with existing configurations in Active Directory, automatically enforcing enterprise-defined roles and policies across all connected data sources.

Secure Data Handling Practices

To mitigate risks associated with storing sensitive data, IllumiDesk employs:

• Data Encryption: Sensitive data is encrypted both at rest and in transit (in transit in this context is retrieving from RAG stores or communicating with third party systems using LLM  ensuring its confidentiality and integrity.
• Multi-Tenancy Architecture: Our RAG systems are designed to support multi-tenancy, isolating data for different departments or teams to prevent unauthorized access.
• Auditing and Monitoring: Comprehensive logging of data access and queries provides visibility into system usage and helps identify potential security vulnerabilities.

Flexible and Scalable Administration Options

IllumiDesk reduces administrative overhead with:

• Centralized Administration for Multiple Knowledge Bases: Administrators can manage RBAC configurations for multiple departments or teams from a single interface.
• User-Centric Augmentation: In addition to shared knowledge bases, users can augment queries with their personal documents, ensuring flexibility without compromising security.

Real-Time Authentication with Agent-Based Retrieval

For third-party integrations, IllumiDesk offers agent-based retrieval workflows that maintain security:

• On-Demand Authentication: Users authenticate directly to third-party systems (e.g., Confluence or SharePoint) during agentic flows, ensuring that only authorized data is retrieved.
• Direct Data Access: The system queries third-party APIs in real-time, avoiding unnecessary storage of sensitive data within the vector store.

These security considerations are highlighted in the illustration below:

Figure 2: securing RAG systems

Conclusion

IllumiDesk’s RAG solution is designed with enterprise security at its core. By enforcing robust RBAC, integrating seamlessly with solutions such as Microsoft Active Directory, Okta/Auth0, and AWS Cognito, and employing secure data handling practices, IllumiDesk's solutions help organizations meet their compliance requirements while leveraging the power of large language models.

For senior technical teams seeking a secure and scalable RAG implementation, IllumiDesk is the partner you can trust. Let us demonstrate how we can align with your security priorities and drive innovation within your enterprise.