--- title: Who's reading my file? description: Three useful tips for finding what files are being opened and by which process. --- [Home](https://mutuallyhuman-pr-605.herokuapp.com/) - [Team](https://mutuallyhuman-pr-605.herokuapp.com/team) - [Work](https://mutuallyhuman-pr-605.herokuapp.com/work) - [Blog](https://mutuallyhuman-pr-605.herokuapp.com/blog) - [Contact](https://mutuallyhuman-pr-605.herokuapp.com/contacts) # Who's reading my file? By [Zach Dennis](https://resources.mutuallyhuman.com/test/author/zach-dennis) on 23 10 2013 *We had just deployed to a staging environment and nothing seemed to be working right. There were no problems though when running locally or in the integrated dev/QA environments. The logs showed there were SSL certificate verification issues, however, the SSL certificate was validating fine in browsers. Something else was amiss, but what?* We did find and solve the SSL issue. It was caused by the customer-specific truststore for root certificates. It was missing the certificate for the certificate authority that had signed their SSL certificate, Comodo. That's not the interesting part though. There were multiple keystores and truststores on the production server not counting the system or JVM defaults. After spending time looking through configurations to see what was being used we decided to take step back and interrogate the system. It could tell us what file was being opened. This got me thinking about three ways that I had used in the past to interrogate the system: lsof, strace, and dtrace. In fact, I even got to learn about a fourth: SystemTap. While I didn't get to use all of these to solve our issue (as dtrace and SystemTap aren't available on all systems), they are good tools to have in your toolbox. Here are three simple ways to use them to find out who's opened a particular file, or, as in our case: what file has been opened. ### lsof `lsof` is a unix command which is for "list open files". It can be used to determine which process currently has a file open. It has a lot of options, but in its most basic form is incredibly easy to use: The above command will show you which process has `/path/to/file` open. Here's a working example: If you close the file or end the irb session and re-run `lsof` you'll no longer see output as there is no process with the file currently open. You can find more information via its main page. There are a lot of options, but at its heart the simple rule is this: if a process currently has a file open `lsof` will be able to tell you about it. This is a blessing and a curse as only being able to interrogate the system about currently open file handles can be limiting. A great list of `lsof` examples can be found over at [http://www.catonmat.net/blog/unix-utilities-lsof/](http://www.catonmat.net/blog/unix-utilities-lsof/). ### strace `strace` is a linux command for tracing system calls and signals. It works by running a specified command, intercepting and recording system calls and signals, and then reporting its findings when the command has finished. The basic usage is this: Now run strace on `sample.sh`: That prints out – unfiltered – all system calls and signals that occured when running the ruby command. It's a bit more than what we care for. `strace` does provide a way to get better signal to noise ratio. We can specify a qualifying expression using the `-e` option to indicate we only want to see `open` system calls: Your output will now look a lot more reasonable and possibly a lot like this: This is pretty close to what we did when looking to see what truststore file was being opened by our process, with one major caveat. By default `strace` won't enabling tracing for child processes. Let's run the above command again and cause our ruby process to fork before reading `/etc/passwd`: This time you'll notice that we're missing the last system open on `/etc/passwd`. We can fix this using the `-f` option which tells `strace` to trace child processes: This will trace forks of forks of forks: One last tip for using `strace`. It's the `-o` option which lets you specify an output file. This can come in handy when the process you start with `strace` ends up being started by other processes (like init.d daemons) and you don't have access to its STDOUT. The following example logs the system calls to `/tmp/files-opened.log`: `strace` is a great tool especially on linux systems since it seems to be installed on many distributions by default. It's not installed by default on OSX or other unix systems as far as I know although there appear to be packages that may be installed. A good list of `strace` examples can be found at [http://www.commandlinefu.com/commands/using/strace](http://www.commandlinefu.com/commands/using/strace). ### dtrace `dtrace`, which stands for "dynamic tracing", is the bees knees in process and systems debugging. It gives you the ability to trace dynamic languages, compiled executables, libraries, system calls, kernel calls, and hardware calls – and does it dynamically. There's a lot to `dtrace` and its power extends far beyond what `lsof` and `strace` can accomplish. There are a few books on `dtrace` ([one](http://www.dtracebook.com/index.php/Main_Page) of which I am currently working my way through). Since the scope of `dtrace` is so vast I'm going to force myself to keep it super minimal. Oh, and you should know, `dtrace` is available on OSX 10.5 and up, FreeBSD, Oracle Solaris, and OpenSolaris. There is a [port](https://github.com/dtrace4linux/linux) to linux but I haven't used it. And by default it requires superuser privileges to run. Here's an example for finding which files are being opened on your system: This will output all of the files that are being opened on your system by all processes. It's quite a bit of information! If only there was a way to narrow it down. Well, you're in luck. With `dtrace` you can use predicates to narrow down the results collected. For example, start an `irb` session in one terminal, and then open a second terminal. In that second terminal, let's find out the PID of the `irb` session: Now insert that PID into the follwoing `dtrace` command and run it: Leave `dtrace` running in your second terminal and navigate back to your first terminal. The one with `irb` running. In `irb` read a file like so: Now go back to your `dtrace` terminal. You'll see something similar to this: Any time your `irb` session opens a file `dtrace` will tell you about it. This is not limited to `irb` either. You can trace any *open* system calls for any process. It's just that in the above example we chose to focus one our `irb` session. Let's revisit the problem described at the top of the post for just a second. We could have used `dtrace` and the above command(s) to help us identify the truststore file that was being used. Unfortunately, the server was RedHat Enterprise Linux which doesn't have `dtrace`. Had it been available it would have saved us time stopping our daemon, modifying the init script to run with `strace`, and then re-starting the daemon to collect output. All in all `dtrace` is very powerful and hopefully this one example provides a little insight into how nice it can be. Many useful one liners for `dtrace` have been shared by Brendan Gregg over at [http://www.brendangregg.com/DTrace/dtrace\_oneliners.txt](http://www.brendangregg.com/DTrace/dtrace_oneliners.txt). ### There's a possible fourth... SystemTap `SystemTap` is like dtrace, but for Linux. It's an opensource project backed by RedHat, IBM, Hitachi, and Oracle that is [available as an RPM](http://www.sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html), but can also be [installed on Ubuntu](http://sourceware.org/systemtap/wiki/SystemtapOnUbuntu). I was using RHEL6 and tried to get SystemTap to work. Sadly, it didn't happen. The necessary `debuginfo` kernel package that SystemTap needs isn't currenlty in the RHEL6 RPM/Yum repository. There's an open bug with regard to this [here](https://bugzilla.redhat.com/show_bug.cgi?id=570352) and an open issue filed [here](https://access.redhat.com/site/solutions/9907). ## Summary It's not every day that you need to find out what low level system calls are ongoing. When you do need to know tools like lsof, strace, dtrace, and SystemTap can make the difference between pulling your hair out and pinpointing the issue. You may need to find out who's got your file in some of the strangest situations, like, in the case of SSL certificates and truststores. *Masthead image courtesy of Paramount Pictures* ### Related Topics - [Uncategorized (106)](https://resources.mutuallyhuman.com/test/topic/uncategorized) - [User Experience (20)](https://resources.mutuallyhuman.com/test/topic/user-experience) - [Ruby (11)](https://resources.mutuallyhuman.com/test/topic/ruby) - [Interaction Design (10)](https://resources.mutuallyhuman.com/test/topic/interaction-design) - [JavaScript (9)](https://resources.mutuallyhuman.com/test/topic/javascript) ### Related Posts ### About Mutually Human Mutually Human is a custom software design and development consultancy specializing in mobile and web-based products and services. We help our clients design, develop and bring to market innovative products and services based on insightful research and strategy aligned with business objectives. We’ve helped Fortune 500 companies, state governments, and startups. - [Home](https://www.mutuallyhuman.com) - [Team](https://www.mutuallyhuman.com/team) - [Work](https://www.mutuallyhuman.com/work) - [Blog](https://www.mutuallyhuman.com/blog) - [Contact](https://www.mutuallyhuman.com/contacts) logo [Mutually Human Grand Rapids: 401 Hall St. SW Suite 430 Grand Rapids, MI 49503 USA](https://www.mutuallyhuman.com/contacts#grand-rapids) [Mutually Human Columbus: 243 N. Fifth Street Suite 300 Columbus, OH 43215 USA](https://www.mutuallyhuman.com/contacts#columbus) +1 616 475-4225 [hello@mutuallyhuman.com](mailto:hello@mutuallyhuman.com) © 2017 Mutually Human