EMC File Activity Monitoring

Note: This is the 4th and final blog of our File System security series. Check out the first three: 1) NetApp File Activity Monitoring, 2) Windows File Activity Monitoring, 3) Challenges with Native File System Access Auditing.

Sign up now for my live webinar “Challenges with Relying on Native File System Logging“. Register now.

In the final post of this 4 part blog series, we will take a closer look at file access auditing on an EMC Isilon file system leveraging native technologies. We will walk through the configuration process, and continue to explore the common challenges faced when working with the resultant audit logs.

Overview of Auditing on EMC Isilon OneFS

EMC Isilon’s auditing capability was revamped with the release of OneFS 7.1 with the ability to enable protocol access auditing cluster wide, and the ability to forward events to a syslog server or third party applications, like the STEALTHbits Activity Monitor, using the Common Event Enabler framework.

Once protocol auditing is enabled on an Isilon Access Zone, file access events are logged on the individual cluster nodes where the client initiated the access attempt. The table below displays all audited actions:

Event Name	Example Activity	Exported via CEE
Create	Create a file or directory	Yes
Close	Close a directory	Yes
Rename	Rename a file or directory	Yes
Delete	Delete a file or directory	Yes
Set_Security	Modify file or directory permissions	Yes
Read	The first read request on an open file handle	Yes
Write	The first write request on an open file handle	Yes
Get_security	Read security information for an open file handle	No
Logon	SMB session create	No
Logoff	SMB session logoff	No
Tree_connect	SMB first attempt to access a share	No

Log files are rolled over to a new file once they reach 1 GB, at which point they are automatically compressed (as of OneFS 7.1.1). One important thing to note is that due to regulatory requirements such as HIPAA, audit log files are never deleted from the cluster unless done manually. This can lead to issues with storage consumption as well as add overheard to the auditing process. While manual cleanup of these files is recommended, this process can cause interruptions in SMB access, forcing administrators to perform these tasks during scheduled maintenance windows. Refer to this EMC support article for audit log cleanup steps.

Event Forwarding

The OneFS 7.1.1 release introduced the ability to forward events to either a syslog server or to third party software via the Common Event Enabler (CEE). It is recommended that protocol auditing is not enabled until the syslog endpoint or third party software is configured due to the nature of how events are forwarded in the case a backlog of events exists. In this type of scenario, it may take the third-party tool time to process the backlog, leaving the result set stale for a potentially extended period of time.

Configuring Protocol Auditing on EMC Isilon

The majority of the audit configuration can be achieved through the OneFS console, but the CLI will need to be used in cases where specific audit settings are required. The recommendation is to leverage a more scoped audit configuration to minimize the system impact.

1) In the OneFS Web UI, select Cluster Management > Auditing

2) Select “Enable Protocol Access Auditing”

3) Add the Access Zone(s) to be audited

For the purpose of this blog post, we will want to add the ability to audit successful and failed read and write attempts.

4) Use the isi audit settings modify command to add the additional events

Viewing the Protocol Access Audit Logs

The configurations applied thus far will store events on each of the individual cluster nodes which can only be viewed using the isi_audit_viewer command unless leveraging syslog or CEE to send the events to third party servers.

Available search parameters are displayed in the screenshot below:

A sample audit entry is displayed below:

Challenges with Interpreting Critical Access Events

Audit Entry Format

Unless the audit events are forwarded to a third party server for further analysis and consolidation, it is nearly impossible to parse them for anything meaningful. Unlike Netapp, these events cannot be viewed through a user-friendly interface such as Event Viewer, forcing users to know exactly what they are looking for before viewing the logs.

In the case where there is a specific investigation where things like date/time frame or target share or UserSID are known, admins can use tools such as grep to refine the scope of their query against the audit logs.

Let’s explore the scenario of a missing file. Let’s say Carrie from accounting can’t find the text file that stores all of the billing codes, which she needs to be able to process expense reports. In this case, she can tell you how she accessed the file (through the Accounting shared folder), and maybe even the name of the file (Billing Codes), but not much more than that. With this information the admin can run isi_audit_viewer –t protocol –v | grep delete | grep Billing which will search the audit logs with these clauses applied.

Here you can see the UserSID responsible for performing this action, along with the timestamp of when this occurred. Depending on the amount of audit data stored, you may benefit from adding filters for the start and end time, assuming you have a relative timeframe for when this may have occurred.

UserSIDs

One huge shortcoming of the audit data provided by OneFS is in the way users are logged. For each event, a UserSID will be logged along with an internal UserID. In order to determine who the actual user is, you can either run a reverse lookup against the UserSID returned, or you can run the isi auth users view command.  In the example above, the event returns UserID=1000038. The command would then be isi auth users view –uid 1000038.

Permission Changes

Similar to the NetApp, the permission change data provided through OneFS leaves a bit to be desired. Let’s explore the scenario where our user Mike adds an ACE to the Accounting folder. We can filter our search of the audit log to the ‘set-security’ event type by running isi_audit_viewer –t protocol –v | grep set-security command.

Again, we can tell what resource was touched and by who, but have no visibility into the added/removed/changed permission.

Denied Access Attempts

The ability to monitor failed access attempts can be crucial to be able to detect early signs of compromise. In both Windows and NetApp, this is indicated in the event through an Audit Success or Audit Failure flag. However, these failed access attempts are not as straight forward to interpret on the OneFS side.

Let’s examine the scenario where Farrah attempts to delete the Billing Codes.txt file although she does not have access. In this case, a series of events with eventtype=Create will be logged. In order to determined whether there were any failed access attempts, you would need to filter based on the ntstatus value that is returned.

The list of ntstatus values is shown in this Microsoft Article. Here, you can see that an ntstatus of 3221225506 was returned, which indicates that access is denied.

There are several pitfalls to the way this data is represented:

• There is no way of understanding what operation was attempted. In the example above, the user attempted to delete a file, yet the event type that is returned is create.
• It is not clear whether access attempts have been successful or not without having intimate familiarity with nt statuses.

Conclusion

In this 4 part blog series, we took a deep dive into configuring file access auditing across various flavors of file systems including Windows, NetApp C-Mode, and EMC Isilon, and explored the quality of the data provided through native auditing functionality. Although each system has its own unique set of problems, there are unilateral challenges with implementing a successful file access monitoring solution solely leveraging native technologies. In order to overcome these challenges, we recommend leveraging third party solutions such as the STEALTHbits File Activity Monitor, which offers a lightweight, easy-to-use, scalable solution that is designed to minimize manual efforts by:

• Streamlining the auditing configuration
• Reducing Noise Events
• Resolving attributes such as UserSID, FilePath, or NTStatus

Learn more about STEALTHbits File Activity Monitor.

Sign up now for my live webinar “Challenges with Relying on Native File System Logging“. Register now.