Posted on September 20th, 2012 No comments
No, I haven’t gone insane but there isn’t a more important word in I.T. than backup and here’s a story with some fixes that may help someone in the future in a disaster recovery where no backups are available.
To give you a bit of a background all of the steps in this post have been tried in a real disaster recovery scenario which I performed on a local business over the past few days. Just to be clear this business was not someone I had worked with before and had called me in on an adhoc basis which is why the next bit I am about to explain was allowed to happen.
When I turned up to their office I got an initial briefing from the Director on what had happened and so the long long day began! There had been a power failure overnight in the office and the server was not protected by a UPS so it had an unclean shut down. Unfortunately this also led to corruption on the RAID array which meant that the server would no longer boot. I then proceeded to ask about backups and the director explained that they just took backups on an adhoc basis to an external USB disk however this had not been done in over a month so the majority of the data held on the external disk was fairly out of date and therefore not of much use. At this point I began to explain that thing’s were not looking good and that we really needed to discuss a proper backup strategy once we had the problem at hand resolved. The director was a little bit protective about the backup scenario and explained that it stemmed back to the cost of backup software especially cloud backups which I was putting forward as the best option. When I provided a price in a debriefing meeting it didn’t go down well however I asked him to calculate how much business he had lost in the 36 hours the recovery took and also how it would impact his reputation and repeat business. It then became clear that the lack of a proper backup had cost him a lot more than the backup software would have originally cost him. This is my biggest point of this post:
“HOW MUCH MONEY WOULD YOU LOOSE IF YOUR BUSINESS WAS DOWN FOR A DAY DUE TO LACK OF A PROPER BACKUP???
CALCULATE IT AND BE PREPARED TO SPEND UP TO THAT ON AN EFFECTIVE BACKUP SOLUTION”
Story and point made let’s get on with how I went about disaster recovery. I was almost tempted not to post the disaster recovery process I went through because it shouldn’t be allowed to get to this stage. To give you a summary this is where we are at:
- Failed RAID array (2 disks in a 4 disk RAID10 out of sync)
- Backup over a month old
- Server not booting
- Business without any files and e-mails
So the first step I took was to download some software called RAID re-constructor which can be found at http://www.runtime.org/raid.htm. This software claims to be able to reconstruct failed arrays however unfortunately as the RAID controller had taken the array offline it was not detected in RAID re-constructor. Some users have reported that they had more success removing the disks from the storage controller and attaching them directly to the SATA ports however the motherboard in this server only had 2 SATA ports and one was being used by the bootable CD I was using to run RAID re-constructor. On the bootable CD it also had another one of their programs called GetDataBack which was able to see a lot of the files on the partition. At this point both myself and the director were relieved that the files were still there so now the task of accessing them without loosing them began.
After a while of searching around I came across a KB article on the Adaptec knowledge base that said to note down all of the array parameters and then to re-initialise the disks. I was very reluctant to do this as if it went wrong I was likely to suffer complete data loss. After more research it did look like this was the best method however to make certain I was not going to loose any data it was off to PC World (I hate to say it but I needed them the same day) to get some hard disks. I then used CloneZilla to clone the original disks to the new ones I had just purchased as a backup in case anything went wrong. I then booted back up in to the Adaptec RAID configuration utility and following the steps below:
- If it asks you to accept or reject changes that have been detected press reject.
- Go to manage arrays and note down the properties of the array. This is mainly the name of the array, stripe size and total size.
- Go to initialise disks and initialise all of the disks that belong to the array. This will say you may get data loss but just confirm these prompts.
- Exit out of the array configuration utility and go back in. This time press to accept the changes.
- Create the array again with exactly the same parameters that you noted down in step 2.
- Reboot the server and see where you are at.
After the reboot I now had the array back and I could see the partitions on the array again however I still could not mount them. I decided to boot into a CentOS live CD to get a better look at the partitions and see if I could mount any of them under Linux as it is a bit less fussy than Windows with errors. Using “fdisk -l” I was able to see my partitions however as they were GPT volumes fdisk was not able to retrieve all of the data. I then used gparted instead which has support for GPT however when trying to retrieve the partition information gparted would just crash. I then started to run other commands on the disk and looking at the output. The most important bit of information I found was from running “pvscan -vvvv” which reported “unrecognised disk label”.
I researched this error for a while and a lot of the message board posts related to MAC and from the posts it looked like most people had been unable to recover their disks and in the end had just formatted. After a lot of reading the conclusion that I managed to make was that the partitions were still there and somewhat accessible however from re-creating the RAID array the volume information was not in the correct block for the system to be able to read it correctly and mount it.
The next step I took was to boot up a CD called “Ultimate Boot CD” (www.ultimatebootcd.com). This contains a utility called TestDisk which I have used numerous times in the past. This excellent utility is able to search disks for partitions and rebuild the MBR and GPT information. If you have reached this part of this post and still reading you probably want to know the steps so here it goes:
- Boot the ultimate boot CD.
- Once booted select the HDD option.
- Select Disk Recovery
- Select TestDisk
- Choose to boot with the default options. It should be the first option
- Once booted into the GUI if you go on to the start menu you should then be able to find TestDisk in one of the folders.
- Once you have launched test disk it will ask you what you want to do with the log. Just press no log.
- You should be presented with a list of disks on your screen. Select the disk which you are having problems with.
- After pressing proceed you will now be asked what the file system type is. Most of the time it will automatically select this for you however if not the chances are you need to select Intel or if your disk is bigger than 2TB then select EFI GPT.
- On the next screen select Analyse.
- You may get a list of partitions on the next screen however as mine were lost I did not. Press quick search at the bottom however and TestDisk will try and find the lost partitions to recover them.
- Hopefully TestDisk will find your partitions and you can then press enter to confirm
- The next screen will then give you the option to write the changes. You need to select this and then exit out of the utility and reboot.
After completing the steps above I tried rebooting again however it still could not find Windows to boot. I then booted back into the CentOS live CD and tried the same commands as before however this time I was able to mount the volume but I could not see any information contained within it. At this point I decided that getting the original system to boot was not going to be possible and even if it was I was likely to spend a long time getting it to that point. Instead I added an extra disk to the server and installed Windows Server on it. Upon boot I then went into disk management and I could see my array there however it was marked as offline. To my delight I was able to right click and bring it online at which point when I went to my computer I could then see the drive.
I thought my day was nearing an end however it couldn’t be further from the truth. Upon trying to access the disk it told me it needed formatting. I went back to the disk management utility and it was now showing the disk as a RAW partition. I ran test disk on this again however it didn’t have any success this time. After a bit of research I found an excellent utility called “Zero Assumption Recovery” which scanned the partition and was able to see the majority of the information that existed before the array failure. This tool will let you run the scan to see what files are available for free and you only have to pay should you then want to recover it. After seeing the information was available I paid for the software which was only around £50 so an absolute bargain for the data it was recovering.
After a few hours all of the data had copied on to the new disk I had added and I began to rebuild the system. A few hours later I had the server back to a fairly usable state that was far better than the copy on the external hard disks they held. It took a couple more days of tweaking to get perfect however my 36 hour ordeal was over and I decided to go home for a nice long sleep!
The moral of the story as the title says is “Backup, Backup, Backup”.
Posted on September 5th, 2012 15 comments
Over the past few days I have been putting together a basic shopping cart website for a client based on NopCommerce. This website also required integration with their existing payments provider Barclays ePDQ. Unfortunately NopCommerce does not have any support for Barclays out of the box and on the forums although there are a few people that had mentioned about publishing a module there was nothing I could find that was openly available. Due to this I took it upon myself to create a basic payment module which I could then contribute back to the NopCommerce community to save anyone else the hassle and time in the future.
Therefore please find the download for the Barclays ePDQ payment module below. This should be installed like any other payment module by uploading it in to the plugins folder. The functionality is limited to just accepting payments, refunds, recurring payments etc are not supported.
This was also my first attempt at making a module for NopCommerce and also working with MVC so don’t expect the code to be perfect but I am open to suggestions if you feel it needs modifying or you want something else adding.
For those who are following this post. Barclays have been in contact with me about publishing this plugin on their website so I have spent a lot of time today updating the plugin. The plugin will now work on NopCommerce 3.0 however not with the single page checkout yet. I hope to have this working with the single page checkout and release the plugin for 3.0 within the next few days.
One of the requirements from Barclays to publish the plugin is to offer some form of commercial backing. In order to fulfill this I will be releasing the updated plugin under my company however I must stress that it will still be free to download it will just be that I will also provide paid support options to help users who are having problems.
I will of course still be offering free advice as I have done here however as everyone so far has understood the free support is subject to my work commitments where as the paid support will be available within 24 hours.
The new module is now available: http://www.vooservers.com/blog/barclays-epdq-nopcommerce-module/
Posted on August 16th, 2012 No comments
If you check my blog regularly or follow it on the RSS feeds you may have noticed a decline in the number of posts over the past year despite me posting numerous times that I intend to post more updates. I just wanted to add a bit of a holding post for now with an explanation. The past year has been an extremely busy one first of all with work commitments which has involved moving all of my companies equipment and offices to a new datacentre in Maidstone. In personal commitments I am also happy to announce that I got engaged in March this year in Bruges to my fiancée Helen so we are now busy planning or wedding and my personal favourite bit.. the stag do!
While thing’s are busy I do intend to pick things back up once they settle down again so hold tight and I will be back with a lot more technical information and also more general information on data centre provision which is something that a number of people have mentioned they would like to know about in my day to day job of running a uk dedicated server company.
Lync 2010 – Active Directory Operation Failed – Insufficient Access Rights To Perform This OperationPosted on August 26th, 2011 No comments
So after a long break in posting due to busy work commitments I am finally back to write some more articles which I have saved up. This one goes back a while to when Microsoft Lync 2010 first came out however it is something that I have come across in a few environments when installing Lync especially in environments where I.T. staff are domain administrators to be able to support their users on a day to day basis.
When editing a domain administrator in the Lync control panel you may be faced with the error below once you press the commit button to save your changes.
To fix this you need to make the following changes in active directory which can be a bit of a pain if you are making regular changes to any domain administrators as the permissions you are about to change will reset themselves after about 5 minutes.
- Go to Start>Administrative Tools>Active Directory Users and Computers.
- Browse to the organizational unit that holds the user account you are attempting to modify in Lync.
- Right click the user and go to properties which will present the following dialog.
- Press the advanced button where you will then be presented with another dialog box.
- Tick the box marked “Include Inheritable Permissions” as shown below
- Press ok to close all of the dialogs and try saving your changes again.
Please remember as I mentioned earlier this fix will only last for around 5 minutes as Active Directory will automatically re-apply the old permissions so if you need to make a number of changes you may want to keep the advanced security dialog open so you can easily re-apply the settings.
Posted on March 9th, 2011 3 comments
While working on a recent integration between NAV 2009 and CRM 2011 for a client I came across the following error when trying to run the default “NAV Customer Card to Account” mapping.
Inner Exception: Requested value ‘Customer_No’ was not found.
This took me a long time to work out so I have documented the fix below however please make sure that you read the rest of this article to determine if your situation is exactly the same. The client’s NAV database I was trying to integrate was from NAV 2009 SP1 however support for the connector was not added until NAV 2009 R2 so the database was missing a few required tables. I had imported these which were tables 5151 & 5150, Codeunit 5151 & 5150 and XMLport 5151 & 5150 and I had enabled the connector from the marketing setup table. This populated everything correctly and enabled the web services which allowed me to start the integration and get to the point where this failure was occurring. After a little research I found that you could browse to the web service URL and view the pages that were being shared on the web service. I did this for an integration that worked using the test database in the same environment and also for my integration which was failing and I found the results shown in screenshots below.
You will see in the 2nd screenshot from the working NAV Server that there is the field “Customer_No” at the bottom. In the NAV server that wasn’t working and reporting the error that field isn’t present and that’s whats causing the problem. If you want to check this on your own server you can bring up a list of the web services by going to http://localhost:7047/DynamicsNAV/WS/Services on your NAV server. You should then see a list of the available web services and you want to look for the one ending “Page/Integration_Ship_to_Address”. If you copy the full url which should be something like “http://localhost:7047/DynamicsNAV/WS/Page/Integration_Ship_to_Address” and browse to it you will then see something similar to the screenshots above and you can check for the existence of “Customer_No” in the web service.
Now the explanation on how to check this and verify you have the same problem has been done let’s get on to actually fixing it:
- Open Microsoft Dynamics NAV Classic with Microsoft SQL Server
- Locate and open the demo database that is shipped with NAV 2009 R2. This should be in your installation media.
- Once open go to Tools>Object Designer
- Find page 300 “Ship-to Address” and go to File>Export. Save the file to your computer.
- Close the demo database and open your own NAV database.
- Go to Tools>Object Designer
- Go to File>Import and find the file you saved when you exported the table.
- Press ok to the dialogs and the table should now be imported/merged with your existing table.
- Refresh the web service URL posted earlier and you should now also see “Customer_No” on the installation that was not previously working.
- Re-run the integration map and it should no longer complain of the error.
Posted on November 25th, 2010 No comments
Recently while working on a client’s exchange server I had the need to setup a user which could send from domains that were not being hosted on the server. The reason for this was due to a legacy application that was using the exchange server to send e-mails to mailing lists for an external company. On the first try I just attempted to authenticate using the domain credentials and send the e-mail however exchange will come back with the following error:
If the user and domain we were trying to send as existed on the exchange server we could simply add send as rights to allow us to send from the address. The problem here however is that the domain is on a remote server so we need to find a way to let us send as any address we want to. After some research over at Microsoft Technet it turns out there is a power shell command “Ms-Exch-SMTP-Accept-Any-Sender” which will allow a user to spoof their from address. To use this command you will need to open up the Exchange Management Shell and run the following command.
Get-ReceiveConnector “CONNECTOR” | Add-ADPermission -User “DOMAIN\USER” -ExtendedRights “Ms-Exch-SMTP-Accept-Any-Sender”
You will need to modify the “connector” and the “domain\user” parts to match up with your own configuration. As long as everything goes ok you can test spoofing the address again and this time it should go through. If you are not sure what the name of your connector is you can run “Get-ReceiveConnector” on it’s own to list the available ones.
Posted on November 22nd, 2010 No comments
When implementing a custom 404 ASP page for a client recently I found that the code of the page was displaying as text rather than the parsed code. Accessing the custom 404 page directly however worked without any problems and showed the correct error document.
In the end the problem turned out to be that you can’t specify the page’s “message type” as file if you are using a scripted page such as ASP as it won’t parse the code. If you specify the type as a url and put in the absolute path relative to the sites root however it will work correctly and parse the code. I have included a screenshot below to show the settings a little clearer.
Posted on November 18th, 2010 No comments
Although it is not recommend there are some times when you may need to change the letter of your Windows drive from C: to another letter or it may have changed by itself to another letter and you need to change it back to C:. In my case after an automated Windows installation my Windows drive letter had come up as D: so I wanted to change it to C:. The reason this had happened was due to a USB disk being connected while the install was taking place so it had labeled that as C: instead. To change the drive letter please follow the steps below:
- Go to start > run and type in “regedit” and press enter.
- Expand the folders to HKEY_LOCAL_MACHINE\SYSTEM\MountedDevices
- You should see something similar to the screen below. We are interested in the \DosDevices\ values.
- If you are in the same scenario as myself and something is already using the C: drive you will need to select it from the list and press F2 to edit the value. You should change this to a drive letter that is not in use which in my case would be “I:”. If you do not have a device using C: you can skip to the next step.
- You then need to find the value which has your current drive and edit that to be C:. So in my case I need to change “\DosDevices\D:” to “\DosDevices\C:”.
- You should now restart your computer and the drive letters should change. You do not need to worry about the boot.ini configuration as this is controlled using the disk and partition numbers rather than the drive letters.
If you get an access denied message while trying to change these values it may be due to the permissions on the registry key. To change this you should right click the “MountedDevices” folder on the left hand pane and go to permissions. You then need to ensure that the “Administrators” group has full control of the folder. If you have modified these settings you should then close the registry editor and re-open it for the settings to take full effect.
Posted on October 21st, 2010 No comments
Yesterday I was working on a new HP DL120 G6 with an Adaptec 2405 RAID controller and I had quite a serious problem with the Grub Boot Loader in that it would always ask me to select the option to boot as though a key had been pressed rather than automatically booting the default kernel. While this is not a problem if there was someone there to press enter when it rebooted as the server was in a datacentre it was not very easy.
Another key note on this is that as Grub started to load it was slow and while showing “Loading Grub Stage 2″ the Adaptec card would have a constant beep for around 10 seconds until Grub had loaded. This initially is strange as usually it loads so fast I never see the text on screen. The second thing was that after selecting the kernel to boot it would then start to beep constantly for about a minute while it perform the initial kernel loading which was also very slow.
I spent a few hours trying to fix this (which is why I am writing about it) changing all sorts of options in the BIOS and on the card itself to no luck. Eventually I went to the Adaptec site and started to read through the new firmware release notes and found that while there was not a specific fix related to the DL120 there were some other fixes related to HP and specifically the ML110 so it looks like Adaptec actively work with HP on problems.
Because of this I decided to download the latest firmware (which at this point is build 17911) and proceeded to flash the Adaptec BIOS. Upon this completing and rebooting the server the Grub Boot Loader started to work correctly and automatically boot the default kernel option.
Posted on May 6th, 2010 No comments
Recently I had the need to find all files on a Linux system containing a particular string of text. To do this you can use the following command adjusting it for your specific needs. This command is very useful in cases where websites have been comrpomised to search for a common string that may have been replaced in all of the files on the website.
find ./ -type f -print0 | xargs -0 grep -l “text to search for”
The above command will search the current directory for the text “text to search for”. You can change the directory you want to search in by modifying the “./” to the full directory path.