Long Term Storage

I have this fantastic server which redirects console output to its serial port and I wanted to take advantage of it.  I went through the process of setting up inittab and grub to work and then I started testing the configuration.  In Windows using Putty, everything worked fine.  As soon as I moved the cable over to the serial port of an OSSIM box, I kept getting this error.

/dev/ttyS0 is not a tty

Googl’ing around returned nothing helpful which is why I am posting this here.

I have a habit of  disabling unused peripherals when I setup a new box.  Since I rarely use Serial Ports, I disable them.  Well.. that was the problem.  The disabled serial port isn’t going to work very well for a serial connection.

Hope this saves someone some time.

Lurking in the depths of the internet is a problem. A problem so large that it is going to cost billions to fix. It will effect every device connected to the internet without exception. Every cell phone, every game console, every computer, every router/modem, EVERYTHING. And it is a secret. Well, not really a secret, just not something anyone talks about.

The problem is the language that the internet speaks, is running out of unique names. Specifically, the IPv4 address space is running out of unassigned addresses. The simply version of why this is a problem is no new websites will be able to be online. It is a lot more complicated than that and will even impact users to a degree, but that is for a different article.

This is where the 600 days comes in. By the estimates of the people who are able to do estimates, the currently unused addresses will run out in about 600 days as of the beginning of 2010. As that day approaches, you can expect all sorts of shenanigans regarding pricing and allocation decisions. It will become much much more difficult and costly to setup your own website/service.

The good news is, there is a fix. The bad news is what I was saying in the beginning. It is going to be expensive as hell and it is going to impact a couple billion devices. The worst part is, you can’t even take steps to fix this yourself right now.

The answer is a new language. Internet Protocol Version 6 or IPv6. It solves the addressing problem for a VERY long time. The current version, IPv4, supports about 4.3 Billion addresses. This is represented by 32bits or 2^32nd. IPv6 supports 340,282,366,920,938,463,463,374,607,431,768,211,456 addresses which is 128 bits or 2^128th. This post does a good job of expanding on what this means.

Because this is a new language, this means all of the devices have to be taught to speak it.  In the vast majority of cases, this is software and could be done for free.  The problem is if you are the manufacturer of such a device, why would you provide a free upgrade, when you could sell a new device?  This is further complicated, by the fact that essentially no one supports IPv6.  The deep insides of the internet do, but the majority of the pieces that are exposed on the internet, do not.  The biggest hurdle is most ISPs (Comcast, ATT, Verizon, Charter, Cable Vision, etc) don’t support IPv6 for their users.  Even if you could go buy replacement devices or upgrades to your equipment to support IPv6, you still don’t have access to the IPv6 Internet.

During the period of overlap when not everything speaks IPv6, we will run into problems of sites only being accessible from v4 or v6.  As time progresses that will go from overwhelmingly v4 to primarily v6 and this transition will take a very long time.  The general masses are going to learn more about networking than they wanted to know out of necessity.  Where did I put that number to tech support?

TL;DR; We have 600 days to make the Internet6 accessible.  After that, things start becoming REAL complicated, real quick.

Note:  This article is meant to build awareness not be complete or thurough.  There are large gloss-overs, simplifications and omissions to keep this from being a book.

Another VMWare issue I started having is Unity mode wouldn’t start.  Click the button, the unity menu pops up and the VMWare window minimizes, but the unity app never displays.

This is also  a common problem with a very simple fix.  Apparently, sometime in the past, I added a line to my /etc/vmware/config file to problem fix or enable some other feature.  This line and Unity just don’t mix.

The fix, per this thread and ultimately this blog post, is to remove the following line from your /etc/vmware/config file.

xkeymap.nokeycodeMap = true

Keep reading the VMWare Communities post for additional details if you added that line to address some key mapping issues.

Like many Linux users of VMWare, I live in a mixed world where not everything I need or want can be done in Linux so I have to resort to dual boots and virtual machines.  Combine that with the sad state of EXT2/3 FS drivers for Windows and you have a situation where you have make concessions just to get by.  For this particular story I am referring to having NTFS formated external hard drives.  I like to run my virtual machines on separate physical disks especially when the physical machine is an under-powered laptop.  So I have my VM on an NTFS external drive, trying to run it form my Linux desktop and I am getting terrible performance.  TERRIBLE.  Like the window graying out cause it is unresponsive and having to kill -9 processes just to get anything moving again.  Looking at the processes the FUSE ntfs-3g driver is pegged at 100% even after killing the vm.

It turns out that this is a known problem in the ntfs-3g driver and is being ‘worked on’.  I say ‘worked on’ cause the problem has apparently been around for a long time.  So long in fact, that it has made it to the FAQ on the ntfs-3g.org website here.  With all of that said, the solution is to simply add a line to your .vmx file.  Note:  This is a per-VM setting, so you will have to do it on all of them.

Edit your .vmx file and add the following.

mainMem.useNamedFile=FALSE

Do note as the state on the website that your /tmp dir must be on a non-ntfs-3g partition or else you could cause additional performance issues.

I have a Ubuntu 8.10 AMD64 Server.  In the syslog, I was getting the following error every few minutes.

console-kit-daemon[5013]: CRITICAL: cannot initialize libpolkit

The error is triggered by the update-modt cron job which runs ever 10 minutes.

This is a bug in Intrepid.  console-kit-daemon requires PolicyKit as a dependancy, but Intrepid (Server AMD64) does not install it when it installs console-daemon-kit.

The simple fix is to install policykit.

sudo apt-get install policykit

Next run of the update-motd job and the error is gone.

So.. it has been a month and a half since my last post.  Did you miss me?  I am guessing you didn’t even know I was here.  Did you?  That’s Ok. I don’t hold it against you.

Where have I been I see you asking.  Well, I went to Asia for a 17 days.  Found out my grand mother had a stroke and has the beginnings of dementia.  Built a new PC cause my File server died.  Reorganized my entire network.  Moved this site to a new provider and moved my old provider to a new box.  Got pulled off base work for my primary contract and added as a part time admin on a new one.  Drank beer.  Played footbag and disk golf.  Hung out with friends and enjoyed loving my girlfriend.

Let me tell you.. it has been fun.  Ups, downs, lefts, and rights.  The Grand Mother and job thing were my favorites for adding stress.

Asia.  Asia was a blast.  Anna and I were gone for 17 days.  We visited Bangkok, Phnom Penh, Siem Reap, Ho Chi Minh City, and Tokyo.  Over 3200 pictures that we are still sorting through.  When I get the pictures put up, I will.. maybe.. go into more details.

Built a new PC.  AMD Phenom 9600 Quad core, 4GB Ram, 650G HDD, Gigabyte GA-MA78GM-S2H MoBo, new case, a few fans, and a lot of time figuring out how to transfer ensure I didn’t lose my MP3 collection.

Because the file server died and I subsequently decided to not replace it with the new PC and instead am using the new PC, I had to reorganize the network.  I repurpose  my laptop for the time being to be an internal DNS server.  I purchased a Linksys WRT54G-L which I will be putting openWRT on eventually.

I run the web servers for ClanBBF which formerly was supported by a non-profit web hosting service.  The owner officially killed the web hosting portion and moved the clan site to a new box hosted by Blue Razor with a few others.  My/This site used to be on that server as my compensation for maintaining it.  The new box is much too slow, so I ponied up for an Ubuntu VPS with RapidXen.  So far, I am very happy with them.  Root 4TW!! 

During this transition to the new box, my DNS got a bit hosed up.  Much to my dismay, my registrar 1and1, was to blame and the tech I spoke with really didn’t know WTF he was talking about.  I was very close to dealing with the PITA that is transferring registrars I was so irritated with them.

At this point, my network is together, all of the websites are up and running, DNS is correct (I think), and my PC is doing fine.  Cheers to that.

Company politics is something most of us are used to and expect.  The only thing that trumps politics is the bottom line.  When the two collide, you can rest assured that foolish decisions will be made.  In simplified form, one part of the company thought they could do the job cheaper so they are taking over the base work.  My team will still be around for projects, etc.  In this change though, I got pushed over to a government contract doing some Sun Access Manager support.  Let me tell you, I now know where our tax money is going.  Today makes 15 days since I started the process of getting a very low level clearance.  If I am inferring properly from the events that happened today, the paperwork JUST got into the database.  It will still be 1-2 more weeks before I have actual access. 

Soon I will be doing some corporate PKI support and that will be good.  Just got word that a project was finally approved so that will be taking up my time as well.  Speaking of.. I should be reviewing a 200 page document right now.

More later.. I have started a list of posts that I need to make.

I guess I missed it.. RC2 was released on the 19th.  As I am writing this, I am downloading the latest and greatest from VMWare.  The first thing I notice is HOLY BATBYTES.  The thing is 532MB.  Last I looked, ESX wasn’t 532MB and it is a LOT more server.  It is a freaking OS AND server.  What the hell are they including in this thing?  Now, I agree, that a lot of that size could be symbols and other debuging stuff since this is an RC.  But jebus, 532MB!!!

Anyway.. the download is complete.  Lets proceed with the update!

Download Update- check
unpack update- check
stop vms- check
get cold sweat from the thought of dealing with the WebUI- check

The install is just like all of the others.  Very straight forward and nothing new.  One nice perk though, it didn’t have to compile any modules for my Ubuntu Server.

First complaint.. it did not keep my certificate settings.  It renamed them, but did not retain what I had already.  So this acted more like a reinstall than an upgrade.  :/

Odd.. it did keep some of my other configuration changes though.

A new ‘feature’ to be a gotcha.. everytime they update the plugin for the remote console, you will have to restart your browser.  Nothing like being forced to restart your browswer because you upgraded VMWare on a different box.  At least the console worked after an upgrade.

You lose another point VMWare.

Here is a CRAZY idea.. you already created a browser plugin.. give me the ability to power-on/off, restart, and upgrade vmware tools from it.  Maybe that would be an acceptable compromise to forcing us to use Apache, Tomcat, and a crappy web interface/browser plugin.  GRRR

On the tools upgrade.. I didn’t have to compile the modules.  I know.. that will last probably 2-3 more weeks till the next kernel patch for Ubuntu comes out.

A new feature – VMWare Sync.  A backup solution of some sort.. it states go look at the KB for more info..  LOOKING…..  Nothing found.. That also brings up another interesting point..

Per the release announcement, this is the final RC and it will contain

The RC2 build primarily incorporates fixes that we have incorporated since RC1 was released on July 1. So, no new features…….

It clearly states during the tools install.. this is a new beta feature.  I think someone is confused.

The damn prompting for a client certificate is still there. WTF, over?

All in all.. a smooth update.  They didn’t fix anything that was apparently broken aside from the Remote Console plugin now works with Fx 3.0.1.  How long till it has to be fixed again?

The number one take away is.. Don’t forget to update your console plugin before you get too far into the upgrade.

For the past 6 months or so, I have had a pretty strong interest in Apple products.  I have never owned one to date.  My girlfriend has a macbook so I spent some time playing with it, seeing what the differences are and getting a general feel for the OS. After this time an paying attention to Apple, their products, and behavior I am going to have to pass on the company. I have never been a fan of their iPod due to the DRM and incompatibility with other devices/software. Now I see that the Apple mindset I don’t like about the iPod is actually Apple/Jobs and not limited to the iPod.

Steve Jobs is a control freak. It is his way or the highway. I suppose for the 95%ers, that is OK. For me, it is my device, my OS, my computer, and my life to control. I do not need someone else telling me what is the best (only) way to do something or what is or is not safe/appropriate for me. The walled garden that is Apple will not grow much further if it doesn’t tear down some of their confinements.

I have been using Google Hosted apps for my mail domain for over 2 years now and I couldn’t be happier with the service.  It is always available and quite frankly keeps getting better.  Now Google had an email outage yesterday and at latest count, there are over 125 news articles about it along with COUNTLESS complaints.

It befuddles me to think that someone has the rights to bitch about GMail being down for two hours.  A service that most and by most, I mean high 90s percentage do not pay for.  It may even be 99% don’t pay for.  How can you complain about a service that offers soo much for free when it goes down for 2 hours or less?  Is your life/business/etc so critical that you can’t not be with out email for two hours?  Best of all, it was only the web interface that was down for two hours.  You could STILL access it by POP or IMAP.

So I have to ask again, WTF are you complaining about?  If your business relies on a FREE BETA application that much that two hours will have a significant impact to you, then you seriously need to reconsider your IT strategy.  It is no one’s falt but your own if you go out of business for, again with emphasis this time, RELIYING ON A FREE BETA APPLICATION for business critical services.  Period.

I don’t care who you are, what you do, how you do, how much money you make, or who you fuck.  You are frankly an idiot if this makes you mad or damages your business.  And you can quote me on that.

After some lively discussion on Macworld’s forums I came up with a few more thoughts on the subject.

It continually amazes me how people will trade control/privacy/freedom for “Security”. As if some other person, organization or company will always have your own best interest in mind. How often to we act in our own best interest, let alone trusting someone else to do so for us?

A couple of quotes I used during my debate? were from Ben Franklin and an old proverb.

Ben Franklin:

“Those who would give up essential liberty to purchase a little temporary safety, deserve neither liberty nor safety.”

The proverb goes:

The road to Hell is paved with good intentions.

I feel these two statement illustrate the fundamental issues with Apple giving itself this much control. Never mind the security implications which I will get to in a moment.

In this particular case, a company, Apple, has decided that it should have the ultimate say on if an application can be installed on a device they no longer own. To take this to a mild example which may have already happened.. The initial release of the App Store included an application called NetShare. This allowed users to tether their computers to their iPhone for Internet access. Meaning users could access the Internet over their cell phone data plan. This is against at&t’s TOS and they requested that it be removed from the store. Now, with this kill switch, they could also remove it from the phones that had it installed.

The implication of this is Apple is acting as a cop for at&t and enforcing their (at&t’s) policies. This opens a lot of questions about liability and privacy in and of itself, but that is another topic. Why does apple get to decide or even help in the enforcement of at&t’s policy on a device they have no authority over any more. They have transferred ownership the device to the user in exchange for money. Rudimentary property rights.

A more egregious example is what if Joe Developer created an app, started selling it on the App Store and it became immensely popular. Apple, in their insatiable need for $, after all they are a publicly traded company with stockholders, decides they want a piece of that pie. So they develop their own version of the application and start selling it. Maybe it is not as successful or maybe someone decides it is not making enough money for the amount they invested, or maybe Apple just gets greedy and wants the whole pie instead of just a piece. Apple throws the little switch and bam! all of those users who had an application they loved and paid for is gone. Since they still want the functionality, they have to buy it from Apple now. And they really have no choice in the matter since Apple controls the gates to the App Store.

Do you think this is a little far fetched? Apple already pulled an app called Box Office for no reason. Any chance that little application might end up in some Apple provided application in the near future?

The security implications are much worse than the potential wrong doing by Apple. The very fact this exists means that someone else can exploit it. What does it take to get something added to that list? What other functionality can that list or maybe another list provide? What will happen when some cracker gets creates an exploit to take advantage of this? Based on my limited understanding of how it functions, it appears it could be a VERY simple task to exploit this hole. If the device does phone home to a URL, then that means the address can be faked with the DNS vulnerability that is out there. If we take Apple’s response to patching their desktop operating system as an example of their timeliness to respond to something critical, then chances are their iPhone hasn’t been patched yet. (I can not find any mention of it.) Never mind the fact they didn’t even patch it correctly. Do a little DNS poisoning, blacklist the application that provides the phone functionality, and no more phone. Or an even better one. Depending on how the blacklisting works, setup the blacklist to kill Internet access and the phone. It goes to check for new blacklisting and loses Internet, phone, and whatever else you can kill with it (Maybe the docking port?). Now it can’t even go out to get a legit blacklist because it has no Internet access. Ouch.

All of that with out even exploiting anything but a simple existing DNS issue. What other problems could you cause by having better knowledge of the system? Can it be trigger by a local app instead of the phone home function? Could a web page initiate the blacklisting through malicious coding?

The bottom line is, this functionality is bad news. No matter what the intentions were, by having the ability to exert this kind of control on a device remotely will always result in someone taking advantage of it.

Back on the 7th it was being reported that a secret URL can be used by Apple to disable apps on iPhone.  Today it was confirmed by Job’s that the Kill Switch does indeed exist.

I posted a nice comment over on Wired’s article about this.
As the old proverb goes.. “The road to Hell is paved with good intentions.” Sure, Apple could have all of the best intentions to “only” use the switch when there is a ‘bad’ app out there. But as it has already been proven, Apple has not been very clear on what it defines as “bad”. Take a look a the I Am Rich app. It did nothing malicious, but Apple didn’t like it so they yanked it.

What about jailbreaking? Will it allow them to kill apps that were installed after the phone has been jailbroke?

Take it a simple step further, what else can they do with this ‘kill switch’? Is it limited to killing 3rd party applications or can it also kill your phone? What about look through your contacts, email and private content?  Is there a limit to its functionality?

Historically speaking, very few companies can be trusted to do what is right instead of what will strengthen their bottom line. Apple is NO different.

By simply allowing this kind of functionality, Apple is opening themselves up to scrutiny, risk, and a HUGE PR problem.

And finally, the biggest reason why this is not and never will be a good idea. EVERY piece of DRM, and don’t be fooled, that is exactly what this is, has been cracked. What kind of controls does Apple have in place to keep this kill switch locked down? How long till Joe Hacker down the street finds it and starts abusing it? If Apple’s response to the DNS vulnerability is any indication, which is has been shown, of their corporate position on security and ability to manage risks, then I would be VERY afraid of the chances it gets released into the wild. It is not a matter of IF, it is only a matter of when. Also, again given their response to the DNS vulnerability, who is to say that this function is not bug free? What if some glitch in the function causes any of the previously mentioned?

This is no different than the police having a kill switch for your car or Microsofts newest Digital Manners Management scheme. The bottom line is, who gets to make the decision and how long till it is out in the wild?

et another item that I do not like about the new VMWare Server 2. The process name. In SErver 1.x you could identify which vm a specific process represented from either a ps or top. It would display something like

/usr/bin/vmware-vmx myvm.vmx

and continue on with some other stuff.  With the upgrade to Server2, the process names have changed and now require extra effort to see which VM is which when the process is running.

My monitor’s resolution is 1680×1050 and this is all I can see of the process line when running top.

“/usr/lib/vmware/bin/vmware-vmx -# product=2;name=VMware Server;version=2.0.0;buildnumber=101586;licensename=VMware GSX Server for Linux;licensever”

Running ps aux shows a few characters less.

This is the full text of the command string.

/usr/lib/vmware/bin/vmware-vmx -# product=2;name=VMware Server;version=2.0.0;buildnumber=101586;licensename=VMware GSX Server for Linux;licenseversion=3.0 build-101586; -@ pipe=/tmp/vmhsdaemon-0/vmxf0587f70a704b094;readyEvent=58 /srv/vmware/host.example.com/host.example.com.vmx

As you can see.. the bit of info that is needed to identify the vm is at the very end.

VMWare, please rearrange the construction of the command that launches the virtual machines to have the configuration file listed at the beginning so we can easily see which VM is which.

NOTE: The easiest workaround is to run “ps aux | grep vmx”, unfortunately, this doesn’t work in top.

I don’t know what kind of management is going on over at VMWare that they can pass this joke of a management interface be shown to the public.  I have had nothing but capital P problems with this POS.

All of the problems I mentioned before as well as the interface just not working.

I was installing Solaris10-x86 to do IdM developement on my laptop.  I kept running into a timing issue where the clock on the virtual solaris hardware is running fast to the degree that minutes were passing faster than a real second.  Several hours of digging for a solution, I decided to try it on my “Brand New” VMWare Server 2 install.  The only good news that came out of this entire ordeal was the timing issue went away.  Though, I suspect that is due to kernel differences vs. software improvements.  I had the same kind of speedy behaviour out of Workstation 6.5 Beta 1 and 2.

Anyway.. back to the Solaris install.. So the install goes smoothly much to my suprise.  It wasn’t until after the reboot that the management interface problems started showing themselves.  There were not specific issues as a result of the VM or Guest OS install process.  It was more the resultant behavior of a system that has a run-away process.  Namely, a VM of Solaris that will not die.

When the CPU/Memory load of the host server was high due to the Solaris VM, the management interface started flaking out.  Little sections stopped responding leaving “loading” notices for minutes as if.. gasp.. a jsp applet timed out and doesn’t realize it!!  Never returned any errors or gave any suggestions for how to ‘fix’.  At one point the system was denying me access to perform functions on the virutal machines though I am explicitly an administrator on the application.  I would stop the VM, it would report it was stopped at the bottom in the little events section but the top section would ‘grey’ out as if it was still processing my request.  These quirks were resolved by either doing a full page refresh (shift+F5) or cycling the management process on the host.  sudo /etc/init.d/vmware-mgmt restart

The biggest issue I ran into though, was at one point I could not get the webAccess compontent to actually stop.  vmware-mgmt stop would say it completed correctly but doing a process list showed it running.

Bit of warning if you have to manually kill the webAccess process.  There is a separate watchdog process that will reload the service automatically.  It acts like it retains the session problems details through the watchdog restarts.  I had to manually kill both the watchdog and webAccess processes at the same time to get them to actually stop.  Sounds a bit like a virus, eh?

At this poing I am ready to throw VMware Server 2 out the F’in window, but I noticed a little line indicating the virtual hardware version.  Server 2 bumps this up to version 7 which is not backwards compatible with Server 1.0x or Workstation 6.  (Maybe 6.5 beta?)

In all fairness to VMWare, I do see what they are tyring to accomplish.  They are trying to unify their interfaces between the products.  I am sure it was some kind of PITA to maintain the seperate apps for each of the different programs and I can respect that.  But for the love of all that is Good and VMWare.. Give Me Back my Linux GUI!!!

Another side note: On the forums I found that the VirtualInfrastructure Client can be used to connect to Server 2 rather painlessly.  The only problem with this is.. it is written in .Net and no one has managed to get it to run in WIne yet.  So.. this basically means I am forced to either A. used the F@#$! POS web interface and all of its issues or B. run windows in a VM or somewhere to connect to the Server with a GUI.