16.3.2014
It is very simple to build an embedded computer for data acquisition today - find an ARM board (for example Raspberry PI, but I like BeagleBone much more - it is easier to make expanding boards), employ hardware specialist and make a expanding board. Install and configure Linux, database, web server, your own application and that's all:
I assumed that the first unreadable SD card was caused by manufacturer fault. If the card is OK, then it should not die in 14 days. But other card died in next two days. Two defective cards? Third card destroyed in one month begins to convince me that the cards were OK and that I should hunt for a bug on other place.
One of my friend show me a big box originally full of CD and now full of destroyed SD cards. When developing some embedded system he destroyed hundreds of various cards. Cheapest ones, branded ones and also industrial cards with price in hundreds dollars. The industrial cards should stand but they did not. The industrial cards dies as quickly as any other card in his collection. He saves me lot of money.
SD card is not designed to be rewritten often. When used in your camera or phone there some 10 MB are usually written once, next time another 10 MB are written and so on. You care about your data in camera when the card is full or when you want to download your shots. But database works with the card significantly differently. There are a lot of small writes to the card but this happens very often. Once you write some data to database, the database engine adds few bytes to some file and than enforces sync.
Only manufacturer knows how the wear leveling is implemented. It is very likely that successive appending records to your database files overwrites every time the same part of SD card. It is miracle that the card can stand such treatment for 14 days!
First thing you have to make is to limit writes to SD card. Typically Linux system writes these information:
I'll begin with the simplest thing: swap. Your device must have sufficient amount of ram for whole system and your applications. There must not be need for swap usage. The best thing is to switch off the swap completely. Computer can unbelievably slow down when it begins to use swap. Hard reset is the most quick solution of the situation.
Linux after file each reading writes the date and time of last reading to information about the file. Each read operation also writes automaticly some data to filesystem structures, every time to the same place on the sd card. This behaviour can be disabled in /etc/fstab:
/dev/sda / ext4 data=writeback,nodiratime,noatime,norelatime 0 1
Move your database to ramdisk. Ramdisk can be easily created in /etc/fstab:
shm /dev/shm tmpfs nodev,nosuid,noexec 0 0
Ramdisk can be found in directory /dev/shm. Disadvantage of ramdisk is the data loss after system reboot. Your application should think of if and it should be able to create the database from a template saved on SD card. Lost data? Sorry, but it's much better to lost data than to go to change dead SD card in system at distance of few hundreds kilometers.
Note to database: I'm used to use database Sqlite. I have read somewhere that the Sqlite is suitable to run on SD cards because it does not overwrite old records but add new records to the end of database file instead. Do not believe that - Sqlite can destroy your SD card as quickly as any other database. And the worst - the recovery time for bigger database (GB) can be very long. So try to maintain your database as small as possible.
There is a problem with logs that usually every application writes its log to specific file. The log setting is specific for the application (typically web servers). The simplest think is not to start the syslog daemon and redirect all other application logs to /dev/null. If you really need logs, redirect your syslog to remote computer over internet.
Disabling wtmp is very difficult. It is process init which writes to /var/log/wtmp. Google does not help much - I found a lot of questions "How to disable wtmp?" but the answer is typically suspicious question "Why do you want it? Only bad boys need this when they want to cover their tracks in system". Fortunately there is one simple solution. After the start up simply remove wtmp:
rm -f /var/log/wtmp
The init creates the wtmp every time the computer starts but when you delete it, the init accepts it and the wtmp is not created anymore. If you want disable wtmp completely, you should probably modify the source code.
After study various sources I realized that the best filesystem for my SD card is ext4. Only few parameters should be changed and the journal should by disabled:
mkfs.ext4 -O ^has_journal -E stride=2,stripe-width=1024 -b 4096 /dev/sda1
I would recommend you to modify your start up scripts in /etc/init.d to check the filesystem every time automatically without administrator's assistance. I had to modify file /etc/init.d/fsck (Gentoo distribution). To the place where the system required my assistance I have inserted line:
fsck -y /
Different distribution's start up script will be different, so I write no details here.
And one important notice: very often it is not important the 100% function. Much more important is the ability of the device to recognize error conditions and recover from them as quickly as possible.
