Data Management Module¶

The Data Management Module is your single, consistent entry point for bringing your external data files into our GMS² platform.

Acting as a universal adapter, this module provides a seamless and reliable bridge between your diverse storage environments —such as AWS S3, Azure Blob Storage, or On-Premises Systems— and our platform. It abstracts away the complexities of different providers and enforces consistent platform policies, giving you a unified and secure way to manage data from private user workspaces to shared organizational disks.

All of this is delivered through a single, stable interface that protects your workflow from external complexity and change.

User Interface¶

This module provides MVC/Razor Pages UI.

Menu Items¶

This module adds an Data management menu item to the root menu:

• Disks: Logical storage unit (private or shared) mapped onto a provider configuration.
• Datas: Your actual blob objects (documents, models, datasets, etc.) presented as manageable files.

DataManagementMenus class has the constants for the menu item names.

Disks¶

A virtual disk is a logical, tenant-aware abstraction representing a storage surface in the Data Management Module. It decouples the platform’s domain from physical storage providers (like Database, AWS S3, Azure Blob).

Each Disk normalizes configuration, capacity rules, naming, upload constraints, and file extension policies, while mapping internally to the actual bucket/container/table.

Why "Virtual" ?¶

It does not imply a physical volume; rather a logical boundary for ownership, isolation, quota enforcement, retention policies, and provider selection.

• A single tenant may have multiple virtual disks backed by different providers.
• Switching or extending providers does not leak SDK concepts into application or web layers.

Datas¶

Your file hub¶

This is where you add files to the GMS² platform. Upload documents, models, datasets and other files here to use them in the system.

• Choose a Location: First, select which "Disk" or folder you want to use for storage.
• Add Your Files: You can upload files from your computer or create new folders to stay organized.

You can download documents, models, datasets, and other files that you own or have been shared with you using a temporary access token that expires after 60 seconds.

File operations¶

The basic operations available for managing files and directories in the GMS² platform.

File sharing¶

Enable controlled delegation of access to individual files (and implicitly their containing directories) to:

• Individual users
• Organization units

Permissions and visibility are mediated by Access Grants rather than direct ownership changes.

All file access is automatically secured by our global query filter, which ensures users can only see data they own or have been granted access to.

Permission Levels

The system supports four permission levels based on flag combinations:

Common Combinations:

• Viewer (Read): Read-only access
• Editor (Read + Write): Can view and modify
• Reviewer (Read + Share): Can view and share with others
• Collaborator (Read + Write + Share): Full access

You can set an optional expiration date/time for the share:

• Leave blank for permanent access
• Select a date/time using the datetime picker
• The system converts local time to UTC (ISO format) for storage

Displayable metadata¶

The Data Management module tracks rich metadata for both files and directories that can be displayed to users.

Module leverages ETags to intelligently manage your data. By identifying identical files, it automatically performs content deduplication to save storage space.

It also ensures content integrity during transfers, organizes data for fast retrieval, and reliably detects when a file's content has changed.

Blob path generation & ETag structure¶

The module uses a smart, hierarchical approach to generate blob storage paths. The method changes based on whether content deduplication is enabled, which is determined by the presence of an ETag.

What is the ETag?

The ETag is the cornerstone of this process. It is an MD5 hash of the file's content, resulting in a unique 32-character hexadecimal string (e.g., a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6).

Path Generation with Deduplication (Using ETag)¶

When a file has an ETag, it enables deduplication. The system constructs the blob path by breaking down the ETag to create a efficient folder hierarchy.

Path Structure:

{disk}/t{tenant}/{hash[0:2]}/{hash[2:4]}/etag/{remaining-hash}.{ext}

Breaking down the ETag (a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6):

• {hash[0:2]}: The first two characters of the ETag → a1
• {hash[2:4]}: The third and fourth characters of the ETag → b2
• {remaining-hash}: All characters of the ETag from position 4 to the end → c3d4e5f6a7b8c9d0e1f2a3b4c5d6

Benefits:

• Deduplication: Any file with the same ETag will have the identical path, meaning they all point to the same physical blob, saving storage space.
• Performance: The hierarchical structure (using the first 4 hash characters) prevents all files from being stored in a single directory, which avoids "hotspots" and allows for faster data retrieval.
• Clarity: The /etag/ folder in the path clearly indicates that this blob is shared via deduplication.

Path Generation Without Deduplication (Using a Unique GUID)¶

When deduplication is disabled (or no ETag is provided), the system ensures every file is stored as a unique blob by using a GUID-based path.

Path Structure:

d{disk}/t{tenant}/{id[0:2]}/{id[2:4]}/guid/{remaining-id}.{ext}

The logic for splitting the File ID is the same as with the ETag, but it uses a unique GUID instead of a content hash.

When is this used?

• For security or compliance, to prevent anyone from inferring that two users have the same file.
• For legal audit trails, where every file must be stored as an independent copy.
• For file versioning, where each version must be preserved as a unique blob.

Viewing access grants¶

Access grants are displayed in the Properties Modal for both files and directories:

How to Access:

• Navigate to the desired file or directory in your workspace.
• Right-click on the item.
• Select Properties from the context menu.

Inside the Properties modal, you will find the Access Grants tab. This tab provides a complete list of all current permissions assigned to the resource, with the following details:

• Grant Type: Whether the access is granted to a specific User or an entire Organization Unit.
• Target: The name of the user or organization unit that has been granted permissions.
• Permissions Level: The specific level of access (e.g., Viewer, Editor, Reviewer, Collaborator).
• Expiration Date: The date when the access grant will automatically expire (if one has been set).
• Creation Time: The timestamp when the permission was originally granted.

Your Effective Permissions

For quick reference, the modal clearly displays your own effective permissions at the top. This summary includes:

• Owner Badge: A special indicator if you are the owner of the resource.
• Your Permission Level: Your current role and access level for this specific file or directory (e.g., Viewer, Editor, Reviewer, or Collaborator).

Data Seed¶

This module adds some initial data to the database when you run .DbMigrator if the IsDiskSeedEnabled option is set:

• Creates a default Standard Disk virtual disk configuration.

Distributed Events¶

This module defines the following ETOs (Event Transfer Objects) to allow you to subscribe to changes on the entities of the module;

• FileEto is published on changes done on a File entity.

Example: Get notified when a new file has been created

public class MyHandler :
    IDistributedEventHandler<EntityCreatedEto<FileEto>>,
    ITransientDependency
{
    public async Task HandleEventAsync(EntityCreatedEto<FileEto> eventData)
    {
        FileEto tenant = eventData.Entity;
        // TODO: ...
    }
}