Archive for the ‘Backups’ Category

On the importance of having a good backup strategy.

Thursday, October 19th, 2017

I’m shocked that Borepatch and ASM826 aren’t on this like flies on a severed cow’s head at a Damien Hirst exhibition. But apparently it falls to me, as the ex-backup guy.

A non-profit organization in NYC called Bronx Defenders wants to study the NYPD’s asset forfeiture records. They filed a request for this information (under New York’s Freedom of Information law) in 2014, and litigation is ongoing.

The latest revelation? Not only is the NYPD saying they don’t have the technical capability to pull the data Bronx Defenders wants…

New York City is one power surge away from losing all of the data police have on millions of dollars in unclaimed forfeitures, a city attorney admitted to a flabbergasted judge on Tuesday.

More from Ars Technica:

…an attorney for the city told a Manhattan judge on October 17 that part of the reason the NYPD can’t comply with such requests is that the department’s evidence database has no backup. If the database servers that power NYPD’s Property and Evidence Tracking System (PETS)—designed and installed by Capgemini under a $25.5 million contract between 2009 and 2012—were to fail, all data on stored evidence would simply cease to exist.

When it was activated in 2012, Capgemini vaunted PETS—which was built using SAP’s enterprise resource planning (ERP) software platform as well as IBM DB2 databases—as a flagship public sector project. The company went as far as submitting PETS as a nominee for the 2012 Computerworld Honors awards. But the system was apparently designed without any scheme for backing up the database or any sort of data warehouse to perform analytics on the data.

Adding to this, the NYPD now actually disputes that the PETS database runs on DB2:

Neil tells me our whole argument is that the NYPD’s database is not an IBM database so he definitely didn’t say that NYPD personnel said “the database is in IBM.” He says he was referring the Petitioner’s expert, not any NYPD personnel. The “He” would be Robert Pesner, the Petitioner’s expert, not NYPD personnel.

Okay. So it doesn’t run on DB2. What the frack does it run on, and why are there still no backups?

(It looks to me like both Backup Exec and Commvault have DB2 agents. But I’ve been out of the business for a while, and can’t tell if those have been deprecated.)

Edited to add: Now the NYPD is saying that PETS is backed up:

Contrary to some published reports suggesting that NYPD does not electronically back up the data in its Property and Evidence Tracking System (PETS), all such data is backed up continuously in multiple data centers.

Which, I guess, is good for the NYPD. But as Ars points out, it isn’t consistent with the statement in court that there are no backups of the forfeiture database, unless that database isn’t stored in PETS after all. That seems like the more likely explanation, but it raises the questions: where is it stored, why isn’t it backed up, and why is the NYPD so secretive about those first two questions?

Lessons learned.

Monday, August 6th, 2012

So…somebody I know was having problems with their netbook running Ubuntu.

The somebody in question decided (for good and sufficient reasons) that part of the problem might be due to them having done several upgrade installs of recent Ubuntu versions which left cruft on the system. This somebody thought the best thing to do was to make a backup of /home, reformat the box, and reinstall Ubuntu 12.04 from scratch, blowing away all the existing data and partitions.

Which they did.

The somebody in question had a MySQL database on the box that had somewhere around ~2,500 records in it. It was a fairly simple database, probably overkill for MySQL: one table, a few columns.

It turns out that MySQL doesn’t store databases in /home. MySQL stores databases in /var/lib/mysql by default, and the somebody in question never changed the default. (This vaguely makes sense if you think about it; after all, MySQL is intended to be a multi-user database, so why would you store databases under an individual user’s home directory by default?)

The somebody in question found this out after blowing everything away. And, of course, the somebody in question only backed up /home.

Fortunately, the database isn’t that important, and much of the data on it can be recovered from older .CSV files that were used to import the data into MySQL.

But next time, the somebody in question is going to backup every damn thing, not just /home.

The somebody in question is also going to try to get out of the habit of making assumptions about where things are stored.