Feb 132017
 

Found a good post on how to use the  rescue mode in Hetzner to setup a Logical Volume that spans multiple drives. This plus vsftpd can help setup Terabytes of space for storage as backup. In any case you should not use Hetzner for anything closely resembling a legit online service, website etc. It is only popular as Seedbox for a reason as they are quick to lock your server out.

Speaking of seedboxes, if you want to setup one here is a good script https://github.com/arakasi72/rtinst It can optionally install Webmin.

For file storage, search and download etc I have not yet found a tool. It’s mostly find and scp

If you feel adventurous I found a tricked out Seedbox setup script here https://github.com/dannyti/seedbox-from-scratch. Does everything and makes coffee.

With Bacula running on the local server I can send snapshots over to Hetzner albeit at a slow-as-snail speed. 30 Euros per month for 5.5TB of space is not too bad. I picked up a system from the their auctions and its quite alright but one shouldn’t expect performance or longevity from such systems. You get what you pay for essentially.

 

Feb 062017
 

Start by setting up a container/VM with ubuntu 12 LTS

sudo apt-get install software-properties-common python-software-properties

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
sudo apt-get install oracle-java8-set-default
echo JAVA_HOME="/usr/lib/jvm/java-8-oracle"  | sudo tee /etc/environment
source /etc/environment
wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
echo deb http://pkg.jenkins-ci.org/debian binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list
sudo apt-get update
sudo apt-get install jenkins
sudo apt-get install git
sudo apt-get install autoconf bison build-essential libssl-dev libyaml-dev libreadline6 libreadline6-dev zlib1g zlib1g-dev imagemagick libmagickcore-dev libmagickwand-dev sqlite3 libsqlite3-dev libxml2-dev unzip
sudo apt-get install redis-server postgresql-9.1 libpq-dev postgresql-contrib
wget -qO- https://deb.nodesource.com/setup_6.x | sudo bash -
sudo apt-get install -y nodejs
npm install -g phantomjs-prebuilt
sudo su - jenkins 

This step with rbenv maybe unnecessary since the jenkins plugins does the same thing

git clone https://github.com/sstephenson/rbenv.git ~/.rbenv
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(rbenv init -)"' >> ~/.bashrc
exec $SHELL

git clone https://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build

#install ruby version if not using rbenv plugin
rbenv install 2.3.1
#Edit this to match github
git config --global user.name "John Doe"
git config --global user.email johndoe@example.com

ssh-keygen and upload to github as required.

On the Jenkins UI install plugins

  • git
  • github
  • rbenv
  • Rake
  • envInjector plugin to populate .envrc equivalent

follow these guides here for project setup

http://www.jianshu.com/p/0c9cbbd6d787 -RVM specific

http://www.webascender.com/Blog/ID/522/Setting-up-Jenkins-for-GitHub-Rails-Rspec RBENV

Postgres related


Configure postgres pg_hba.conf to match your database.yml. Contrary to popular belief you dont need to copy any database config

Using 9.5+ version

sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" >> /etc/apt/sources.list.d/pgdg.list'

wget -q https://www.postgresql.org/media/keys/ACCC4CF8.asc -O - | sudo apt-key add -

sudo apt-get update 

sudo apt-get install postgresql postgresql-contrib

Despite all the guides, much of Rails on Jenkins is trial and error. Setting up a new project is relatively easy as per the linked guides but YMMV.

In addition I have had to edit code to make it Jenkins friendly and enabling JUnit reports in rails via Gems is very useful.

Note that installing a single instance of postgres on the Jenkins master is alright for small local developments but not useful for large parallel builds as you are essentially on one DB. I hope this is good enough for anyone to get started and help improve upon the setup.

Possible Further improvements:
Use containers for postgres, build vm
Use EC2 or Azure Vm plugins
use Jenkins Pipeline

Dec 272016
 

Finally got a metal storage rack that I will use as Rack space. About US$99. Finally so much space. Comes with removable half panels, just in case. It’s the same sort I have in my storage room. Why did I not think of this earlier!

metal rack for servers/devices

Since I was unplugging everything to get the rack installed, I took the plunge and installed that 2nd Xeon CPU on the workstation and it went without a hitch. This is the first time I have actually installed the CPU myself. I did try earlier without a heatsink and it was overheating so the system would not boot. The BIOS logs accurately pointed as much, with the heatsink, no complains. Cleaned out the system with compressed air and a vacuum cleaner to catch the dust.

Got a lot done for Christmas holidays I must say.

It tooks months for me to get my Second Intel(R) Xeon(R) CPU E5420 CPU for the used Workstation I bought an year ago. There was some hassle and expensive shipping to get the Heatsink as the stock heatsink would not fit the Dell workstation spec. After all the upgrades to this beast of a Dell t5420 Workstation is finally complete.  My final server spec is

Intel(R) Xeon(R) CPU E5420
32 GB RAM
2.5TB HDD
4TB external powered HDD
2TB 2x enclosure, powered disk cloner that doubles as USB drive
Edgemax Pro 3 Gbps router and Unifi pro wireless (next in line for network upgrades)
1Gbps down and 500mbps up fiber internet connection
excludes my desktop and peripherals.
Runs a decent Apache Storm cluster, mirrors OSS projects and chomps on AWS Kinesis streams while deploying to Azure app services.

There is no hyperthreading support unfortunately on this old Xeon and the 2 Disk LVM still suffers from IO issues. It could be the limits of 2009 SATA. I might have better luck with external storage for things like my Raspberry mirror. I have been very happy with WD My Book which is externally powered and has been my offline backup for almost 4 years now.

Wire management is going to get easier. Plus the Eubiq port is amazing for power supply. See picture

Eubiq power strip, still need to manage those cables.

It doesn’t sit on the floor so it’s gonna stay a lot cleaner and quieter. Once I am sure where to place what i an do some cable management.

Full blast LXC containers manage and host all my apps. Maybe I will try Docker at some point.

The workstation is heavy and I forgot to take picture of the installed dual Xeon CPUs. Maybe next time when I upgrade the disks.

Complete setup of my home workspace, much neater with everything else on the rack

my workspace, those cables need managing

Dec 252016
 

Save time and money. Go Backblaze. Crashplan sucks, terrible speed and it has not improved in years. They are full of excuses about it though so they are unlikely to address it in any correct way.

I am using both and going to ditch crashplan once subscription expires.

Keep it up Backblaze.

Dec 142016
 

I have been using du and df with a bit of other bash scripts to gather disk space usage information on my linux servers. I need to clean up disk space often specially when I am reaching 70% usage just to be sure I will really need that next upgrade. On Windows we have WinDirStat and for KDE there is KDirStat (same tool ported to Windows) but for headless Servers GUI is not an option.

I have come accross the command line tool called ncdu that can be installed from the regular Repository. Just yum or apt-get this tool.

 

You can get per directory usage sorted by space taken like this

ncdu <root-path-to-use>

You can even hit d to delete directories and files recursively. None of that xargs weirdness to handle large number of files. It takes care of that.

 

Give it a spin.

Oct 302016
 

I was having this terrible problem that prevented use of my home server. I was starting to regret many things like buying a refurbished Worskstation, the refurbished memory DIMMs or that fact that I chose Centos7 over Ubuntu which I use for development.

A careful and determined Google Searchathon revealed the issue. My reason to persist was that other than disk IO causing feezing there was no issue with the system. I needed to RSync all my backups from the servers locally. Backups cost me over 500US$ per year. I wanted to save some of that money by atleast getting rid of server snapshots that cost 240US$ per year.

 

Ok back o the problem:

“CentOS or any distro appears to freeze and become completely urecoverable except for hard reset when heavy Disk I/O task is performed for example like copying files over CIFS, RSync or or any heavy read writes.”

I don’t even think it was massive I/O as many blog posts suggests that would cause this. Just about anything would cause it. Pretty stupid if you ask me that after 6 years the problem is still packaged and shipped to everyone.

The search led to a solution from 2010! This was a blog post and a Stack overflow post linked to blog post.

Gist is, it’s your IO Scheduler, the default one is [cfq] more on this below.

I dont immediately modify my server just because of one ServerFault.com answer or Blog entry so I checked the official documentaion at Redhat https://access.redhat.com/solutions/5427 It does appear to be the case that my system is using [cfq] as well.

I switched to deadline as noop is basically no scheduling, useful only inside VM and Containers.

 

This is the blog post for reference:

https://blog.vacs.fr/vacs/blogs/post.html?post=2010/08/28/Solving-Linux-system-lockup-when-intensive-disk-I/O-are-performed

TL;DR

Change your IO Scheduler from CFQ to something else.

 

Check which scheduler is used by disk. LVM or not is not relevant here. Use pvscan to find your disk labels.

$ cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]

$echo deadline >  /sys/block/sda/queue/scheduler

Above command applies it immediately. The settings are gone on reboot.

Do this for any or all drives. To persist across reboots refer to Redhat documentation linked above. You can use /etc/rc.local as well.

Sep 232016
 

For the first time I have started using LVM on my home server and things have been alright for an year. I am able to resize, replace disks , expand size of my mount points or move mount to a new disk without any trouble at all.

While its not relevant I am using CentOS 7.

I have had to force reboot the server a couple of times over the year and woefully XFS complained of corruption 24 hours ago and it was time to figure this out. I kept dropping into emergency mode upon reboot. There is also another reason for this problem. I have bad memory, I bought some refurbished FB-DIMM DDR2 with ECC and ironically not all of it is good.

Anyway TL;DR

Filesystem : XFS

Boot with rescue CD/USB whatever and dont mount anything

Using LVM? yes: Try with lvscan to confirm

I  run xfs_repair /dev/sdx1 and it complains of bad superblock and it also fails to find a good secondary superblock. Something like `Sorry, could not find valid secondary superblock
Exiting now.`

Wait! I need to repair from the Virtual Volume

start with

vgscan -v --mknodes

find and create /dev/xxx/ nodes for the lvm.

Now activate the volumes (it does not mount them)

vgchange -a y

Now see what volumes are active with another lvscan

I have /dev/centos/root which is what I need to fix. It is an XFS volume so this time I do

xfs_repair /dev/centos/root

bunch of output follows…. repair is done.

Lets mount and test

mkdir /mnt/test && mount /dev/centos/root /mnt/test

Looks good?

Unmount and reboot without rescue CD.

 

 

Jul 182015
 

Earlier this month towards the end of June I lost my main drive on the desktop.

surprised-gasp-l

It took me 2-3 days to figure out that it is not possible to boot the drive. To avoid further harm on drive I used my Second drive to boot my Windows 8.1. That was a stupid move. The windows was working again but I had lost all data on my second drive. I was certain that since I had two cloud backup providers thing would be ok. Unfortunately Backblaze which had all data except for “Program files”which it does not back up , I had no way to download it. There restore software does not work. Only way is to ship the data over.

Crashplan was much better. The application can restore and they have a Hong Kong location which is close. However it will take 3 weeks. I am now at last week of recovery. The other problem is that Crashplan backs up very slow so I did not have all the data. I will get bits and pieces.

I have the damaged internal HDD which I am trying to power on with external connector. It doesn’t seem to work. I will need to get a drive bay and try again as it did have an old copy of my data drive.

While I was in Tokyo last week, I managed to get a good deal on Samsung 3d SSD 1TB as well as Western Digital 6TB Red. I have hooked them up. I will be using the 6TB to copy all the recovered data from different sources. Then I need to compare and get the latest version out for restore. The SSD will replace my original Data drive.

 

If the above was not enough one of my main Servers had 2 out of 3 drives fail (hardware fault) earlier this week. Since I was travelling I could not investigate. Despair!


neutral-whyyyyy-l

 

All hope was not lost.

This is a 3 disk Raid5 with mdadm software raid and last night I started to investigate. I was able to boot to rescue mode and found that atleast 2 out of 3 drives can be read by the OS. Only one was completely unreadable. I asked the Hosting provider to replace only one of the two faulty drives (the unreadable one) for now so I can reassemble the Raid5 and start recovery. The recovery started successfully after a couple of failed attempts with mdadm. With forced mode I was able to reassemble the Raid. cat /proc/mdstat shows that recovery is in progress.

It has been over 12 hours now. I believe another 3 hours should complete the recovery and I can mount the Raid disk to check. However I will not be using the current volume yet at all. The faulty drive may fail further and data loss can occur.

Now the plan is to replace the 3rd drive once Raid has completed recovery. The recovery will start again with the 3rd new drive. I should be able to get the complete system up in another 24 hours. Then I will start a new Server in Raid1 with enough space to pump data across to the new Server. I will need to download all of the 2.5TB of data from the new host as my first backup and then setup Rsync.

In the meanwhile, I hope I can get all my development environment, my extremely bleeding edge modded Skyrim back on my desktop T_T

What a month!

 

 

 Posted by at 5:38 pm
May 312015
 

DNS hijacking or plain host hijacking is pretty common these days. It’s a safe low-tech attack to get username and passwords of key hosting accounts. Recently I received an email that was 10/10 where quality of Phishing emails applies. None of the broken English words, in fact see the screenshot. Well written with an official disclaimer with an address to boot.

Decipher Inc phishing email screenshot

Scam phishing email

Few points to note here of why this email is well crafted but still a scam.

1. Yes I am a customer of Linode. How do they know?
Normally these emails are crafted using WHOIS information. However my Whois record is private for the domain in question. This is where it gets interesting.
The Nameserver is with Linode and it is a good guess that if the name server is on Linode (ns*.linode.com is pretty obvious) then I am a customer of Linode. It grabs my attention.

2. This email did not go to spam folder. Private Whois lists a proxy email. When they sent the email to Whois contact it reached the proxy email who forward to me. The Gmail spam is pretty good at picking scan emails up but this one passed right though. If you see the screenshot they have taken every precaution to not use common words like “account” that alert Gmail. Linode ofcourse has my email so they would not send it to my Whois proxy email. Another reason you should get Private Registration

3. The database is not updated for atleast 3 months. I am not using the Linode Nameserver for this domain anymore. Atleast for the past 6 months. It would seem that the data is mined and collected.

4. Decipher Inc. This is indeed a legit company and offers surveys like survey monkey for free. It is not as blatant as free email sign up but ofcourse it must not be hard to get. You request a free demo and they give you a test account. That’s it. The email ends up looking perfectly Legit.

* Google also allows you to create surveys but you cannot add text boxes for password field and will prevent it from being created with name “password” AFAIK.

I emailed Linode support about this. They confirm that this is happening and they are aware of it. They claim SOA records from DNS are being used but in my case the SOA record has no email address. I believe they use NS record for sure to know if the domain is on Linode and that’s about it. Emails are still sent to Whois Contact blindly.

I hope the Linode will send out an informational update before they get overrun with confused noobie admins.

Apr 092015
 

Ever since I found Backblaze, and I don’t know why I did not find it earlier, I have been quite happy with money well spent. I currently have 4 USB drives apart from 2.5TB internal space. This totals to a lot of space and I only use 4Tb of it. This is my home setup. I have tons of VMs and development stuff lying around. I recently lost a fairly new Western Digital Drive of 2TB for no good reason. It just died…. like physically. Taking 1.5TB of VMs and snapshots. Terrible day as I spent 2-3 hours opening the drive up and confirming it was completely dead. I should have seen the signs…

I decided online backup was the way to go. While slow atleast it would keep my data recoverable without adding more drives. I looked at Amazon S3 as my first Choice. However with Terrabytes of data the cost would become VERY VERY prohibitive. I don’t mind paying but the difference between buying drives and backup has to be equal or less to be economic. I could just as easily built a raid 1 by buying lots of cheap drives and putting them into a NAS for backup.

After (a few) quick Google search revealed BackBlaze. A lot of Sysadmins around swear by it. I took a look at what these guys do and this post and knew they try very hard to make data backup affordable and reliable.

Their plan amazed me. US$50 for 1 year of unlimited backup per system and it includes attached drives. Secondly the ability to ship drives of your data back to you. While I have 1GBps connection it would still be slow to get a few Terrabytes from across the world and it is not going to be healthy use of time.

I downloaded the trial. I had some issues initially getting it running because the interface isn’t very intuitive but once you know what’s where it’s pretty much on it’s own as it starts its first back up. So off I want happily to sign up and backup everything.

Unfortunately the upload speed was painfully slow…. somewhere in the region of 200-500KB/s at best. It would take a whole 3 years to transfer 5TB of data if not more. I emailed support to ask if there is option for multiple parallel upload since their application uses only ONE upload thread. They replied back that they would “soon” be releasing the multithreaded upload/download. I waited and 3 weeks later I was still left with over 90% of the volume of my files. However Backblaze was smart enough to keep the larger files for the last. The number of at-risk files were a few thousand. I usually keep my PC on at all times but I dont mind restarting now and then due to forced Windows 8 updates.This time I have had it on since Backblaze for almost 21 days non-stop. Something is better than nothing I say.

 

I was hypothesizing (read:daydreaming)  upon what could change in my life over the next 3 years as the first backup actually started to come to a closure…. what would I be doing on the day when I  discover it actually completed. Perhaps I would celebrate and then find out a few more files have been added to the never ending list…. or may be Joe from support was telling the truth

I hate their site for finding any new information so I googled for the answer once again hoping someone made a workaround…. I could not believe the search result… It has the word “Multithreaded”. In my entire programming life , MT has not made me as happy as it did now.

so the GOOD news! The latest version of Backblaze enabled multi threaded uploads. So despite the physical distances between my BackBlaze and my local desktop, the conspiring bast**d of a  RTT I could now upload in parallel and use the max of my 500Mbps upload speed. Well anything better than the 2Mbps I was getting normally anyway 🙂

Excited. I could not wait and installed the latest version. Sure enough the option was there as per their pitch. With one thread my transfer was shown as 2.35 Mbps which I can easily confirm with other tools and my UBNT router.

With the thread set to 4 backups were already Flying. The file names whizzed by in the Backblaze control panel. Still small files though as the program had found new tiny useless fragments of my entire Raspberry Pi mirror to backup again. Who cares anyway. The only  I could tell if the speed was actually being maximized was when large files would be transferred. I was already seeing an improvement by 10x times

I am currently able to get a consistent 25-30 Mbits/ Sec.  The next step is only to contact my ISP MyRepublic and get then to do a better job of this. A Lower RTT could mean the world of difference.