Wonder what these locations might have in common?
- Bishkek, Kyrgyzstan
- Nuernberg, Germany
- Department of Mathematics, Tehran University, Iran
- Yerevan, Armenia
- Madrid, Spain
- Mannesmann, Germany
- Bievres, France
- Sandanski, Bulgaria
- Kuwait City, Kuwait
- Nuclear Physics Institute, Moscow
- Cheshire, UK
- Tabriz, Iran
- Peace Palace Library, The Hague
Here’s one connecting thread: a computer (or computers) from each of these locations successfully logged into our proxy server in the past week and began accessing databases we restrict to Mason students, faculty and staff. No, it didn’t take significant hacking skill but I’m getting ahead of myself…
I first became aware of the problem late last week when a bot from one of our database suppliers sent me an email saying that significant downloading activity was occurring on their database from an IP address on Mason’s network:
However, recent usage made of this service from your institution exceeds what is regarded as normal and reasonable. This activity was isolated to one host identified at IP address 129.174.55.245 on April 27, 2007. Many of the requests were sequential and systematic–that is, 251 Article requests in Physical Review Letters in addition to 8 requests from various journals were downloaded consecutively and within one hour. Access from the IP range 129.174.55.245 has been temporarily suspended.
No surprise when I found that the IP resolved to our proxy server. Heavy traffic coming through that box isn’t unusual (after all, it exists to funnel the traffic of 30,000+ authorized users to restricted databases when they’re off-campus). Of course, those 30,000 potential users have thousands of possible destinations (databases, ejournals, etc.) so it is uncommon to see any one resource getting hammered from multiple users simultaneously.
I went through the log files and found the series of events that triggered the warning. A “whois†on one of the IP addresses resolved to The Department of Mathematics, Tehran University. Hmmm…
Our faculty travels widely and it’s not uncommon to hear from them in locations around the world but an academic relationship with Tehran University seemed oddly out of place. I checked a few other log entries around the same time (a few minutes after midnight, GMT-4) and noticed that quite a few were sessions for this vendor and they resolved to truly “off campus†addresses: Bulgaria, Iran, France, Spain, Russia, Kuwait, Armenia, Germany and so on.
But wait, I know Mason aspires to someday become a world-class research university. Could it be that it was already happening and I was just one of the first to realize it?
Nah. This was just information piracy and I needed to figure out a way to stop it before Mason’s students began to suffer (the temporary suspension of the proxy server’s IP lasted only an hour but database vendors will block access for an entire campus if they have reason to believe they’re being exposed to unauthorized use).
I began by putting RejectIP statements in our proxy server’s configuration file, blocking access from any of the IP ranges I was seeing in the log. A quick fix that would stem the immediate flood but given the size of the net and the random way in which IP addresses are assigned, it wouldn’t scale as a permanent solution.
We use the campus LDAP server to authenticate access to our proxy server, a change we made last year to improve security and simplify authentication for our users. My working assumption was that someone had probably posted their username and password somewhere on the net and thanks to information osmosis at internet speed, it was out in the wild. But to fix this, I needed the right username.
I spoke to the fellow in our computer center who manages the LDAP server and got an extract of the logs for the time period in question. Too bad…our timestamps didn’t match up (I’ve since setup the ntp daemon on the proxy server to improve time synchronization). Not only were the timestamps out of sync but each log reported only pieces of the total transaction. I just couldn’t cross-validate enough information to get the username.
I googled “mutex.gmu.edu” and the word “login” but didn’t turn up anything of use for this particular intrusion. I did find pages like this one, and this one, that posted now obsolete information on sneaking into our system with institutional ID numbers (an approach we abandoned last summer when the campus LDAP came online). But after all that searching I still couldn’t find out what login credentials these invaders were using nor could I find how the information was being passed around.
Believing the data I needed must either be logged somewhere by our proxy server or visible briefly if I knew where to look, I sent an email to support at usefulutilities.com. Chris Zagar (creator of EZProxy) responded within the hour, pointing me to this page:
http://www.usefulutilities.com/support/example/securing.html
I implemented a few auditing commands via statements in ezproxy.cfg and in no time had the answer to my question:
summer
Turns out there’s a website set up to promote the summer session at Mason and when you have a question about the program, you send an email to “summer@gmu.edu.†Thus, “summer” has an entry in our LDAP database. That would be OK if there were a strong password on the account. Surprised to hear the password was also summer ? If so, you’re not wielding the level of social engineering skills commonly seen in today’s hacker.
Summer’s been blocked as a user on our EZproxy server (meaning it doesn’t even bother to query the LDAP server for authentication) and our campus IT security people convinced the “summer†user to go with a stronger password. And our pirates? It’s been a few days now but they keep trying. Here’s an excerpt from login activity log (courtesy of one of the audit switches in ezproxy.cfg) from this morning (have blacked out anything that’s not a “summer” session):
Summer’s still trying to do research…
Let me close this post with three points:
- If you run an EZproxy server, search your machine’s name and IP address on Google and Yahoo! and see if you find pages detailing how to get in. You may be amazed at the number of exploits you find.
- Read the page on securing your server and turn on the audit switches from time to time to see what sort of activity you’re getting.
- Remind people that in an interconnected, easy-signon environment, simple passwords can lead to all sorts of serious problems.