Virtualization, Storage, Community

Failed sending update file: (NETW_FRMW_WIN_R299290.EXE) to iDRAC - Details: The update package (NETW_FRMW_WIN_R299290.EXE) is not supported via 1x1 update feature. Use the repository method to update this device. This error can also be seen if package is not named according to Dell naming standards.

(download)

Click here to download:

downgrade-firmware-on-an-dell-esxi-host-CzwFwDfCEvkIFnzGDuFh.zip (171 KB)

I racked my 2 new EqualLogic PS6100E SANs today, furthering my belief that storage is sexy.

I had the privilege of attending an HP Cloud Tech Day this past week in Houston, organized by Ivy Communications. Tom, Chris and Halley did a great job gathering some pretty cool and smart bloggers and thinkers to hear about and give feedback on HP's cloud offerings and aspirations. The list of attendees were:

Cleaning up Active Directory is a necessary evil. You need to stay under your CAL count and it can be difficult to figure out which computers (or users) have not logged in to the domain recently.

Windows Server 2003 introduced the lastLogonTimestamp attribute which replicates between all DCs in the domain. Now, this isn’t a real-time data, in fact it can be up to 14 days behind the current date, depending on your domain settings. If you want that, you’re going to have to get yourself a good syslog server, but for general cleanup and auditing purposes it works great. You can read more about this attribute on Microsoft's TechNet Blog.

I’ve written a couple very simple PowerShell scripts that will 1) search the entire domain for all computers with a lastLogonTimestamp before a certain date 2) return a computers lastLogonTimestamp in a human readable local format. It’s not so easy to just go out and get the time stamp, because the format that AD stores it UTC (GMT) format, so it needs some converting to human readable, which my scripts do.

get_lastLogonTimestamp_from_host.ps1

# Gets host and lastLogonTimestamp in UTC of specified host

# get Name

$hostname=Read-host "Enter a hostname" 

#grab the lastLogonTimestamp attribute

Get-ADComputer $hostname -Properties lastlogontimestamp |

#output hostname and timestamp in human readable format

Select-Object Name,@{Name="Stamp"; Expression={[DateTime]::FromFileTime($_.lastLogonTimestamp)}}

-----------------------------------------------------------

get_stale_hosts_lastLogonTimestamp.ps1
 

# Gets time stamps for all computers in the domain that have NOT logged in since after specified date

$time=Read-host "Enter a date in format mm/dd/yyyy"

 

# Get all AD computers

Get-ADComputer -Filter * | 

 

# Make sure to get the lastLogonTimestamp property

Get-ADObject -Properties lastlogontimestamp | 
 

# lastLogonTimestamp - date specified is less than zero, outputs it to a CSV file is working directory

where {(([DateTime]::FromFileTime($_.lastlogontimestamp) - ([system.datetime]$time)).totaldays) -lt 0 } | 

 

# Output hostname and lastLogonTimestamp into CSV

select-object Name,@{Name="Stamp"; Expression={[DateTime]::FromFileTime($_.lastLogonTimestamp)}} | export-csv .\all_old_computers_timestamps.csv

These are two scripts that I use pretty often when I'm trying to determine if I should disable/delete computer accounts in AD. Hope if helps someone else.

Today, Dell released Update 1 to the 1.0.1 version of their Management Plug-in for VMware vCenter. The biggest highlight among the fixes and changes would be the added support of ESX5 (vCenter 5). If you're currently running the 1.0.1 plugin under a vCenter 5 environment (which 'works', just not in a supported kind of way), you'll need to unregister and re-register the Dell Management Plugin after upgrading (see the Release Notes for all issues/resolutions). 

One of the major changes from the original 1.0 to the 1.0.1 plug-in was the promise that updates to the appliance/software would come as an RPM patch and not tied to re-deploying another OVF. I'm glad to report that this worked wonderfully. You can find full instructions in the Dell Management Plug-in for VMware vCenter User Guide (page 41), but here's the quick and dirty:

Ok, I've had the plug-in running for a few weeks and have gone through some of the primary functions of it (firmware updates, inventory, monitoring, warranty retrieval, create hardware profile for deployment)

I'm not going to go through the initial setup, that's been covered pretty well on DellTechCenter.com.

Here are the claimed major functionalities with my notes as far as day to day usage as well as some miscellaneous thoughts at the end.

Deep-level detail from Dell servers

The level of detail here is quite good; much deeper information and more clearly laid out than the basic 'Hardware' tab in vCenter. But what stands out to me is the efficiency of not having to rely on another tool, be it OpenManage, iDRAC, IT Assistant, etc. I spend a lot of time in vCenter and it's fantastic to not have to leave that for another program.

The amount of detail for hardware information is ridiculous. All of this information is available if you have the Enterprise iDRAC in your server, but to be able to get the serial number and manufacturing date of your RAM in the same place that you can check your warranty status is just beautiful.

Deploy BIOS and firmware updates within vCenter

This is a wizard based process that requires you to have a CIFS or NFS repository, which the initial setup walks you through for configuring. I've found it pretty straight forward, easy and quick. Well, the wizard is quick. While this feature is fantastic and works very well, the actual upgrade, however, takes quite a while. The server goes through multiple reboots throughout the process. After the updates are downloaded to the repository, the server is automatically put into Maintenance Mode and then reboots into an EFI environment to do the updates. After each update, the server reboots and re-enters the System Update environment to continue with the next update (firmware/bios). If you attempt to perform many updates at once (NIC firmware, BIOS, HD Firmware, etc.), be prepared to wait.

Build hardware and hypervisor profiles and deploy any combination of the two on bare-metal Dell PowerEdge™ servers without a preboot execution environment (PXE)

This is accomplished through the magic of the combination of the LifeCycle Controller and the iDRAC. While I've built the profile which seems very straight forward, I've yet to be able to test this (spare Gen11 PowerEdge servers are hard to come by, though if one were donated, I would not complain). Although I have a new server coming to replace an out of warranty cluster host that I was planing on testing on, I found this little nugget in the Admin Guide

The system needs to have a Virtual Disk for installation of the OS.

The Plug-in will not install the hypervisor to an internal SD card.

Bummer. This is the standard config for my cluster going forward. No Hard Disks. My great hope is that this is resolved in the next version. If not, this is a huge feature and potentially massive time saver that's not available to me.

Automatically perform Dell recommended vCenter actions based on Dell hardware alerts

The Plug-in adds a whole host of new Dell server specific alarms to vCenter. These range from power consumption to OS driver version monitoring. If something critical enough happens, say a single power supply in a dual power supply system dies, the Plug-in will automatically put the host in maintenance mode until the issue is fixed. This can theoretically save you from encountering an HA event, which, while cool, is never fun.

When I first installed the plug-in, I was immediately alerted to the fact that I was running on a quite old RAID controller driver. Handy.

Receive proactive renewal alerts from Dell before your warranty expires and access the Dell hardware warranty page online 

I've always been bad at doing this myself. It seems easy to track on my own, but we're all lazy in some areas, I guess this in one of mine. So, thanks, Dell, for enabling me to not have to come up with a better solution on my own :) I have yet to receive this because the server I'm testing on still has almost 1500 days of warranty left. But I see the link to click to renew it if I like, and its status is in the Overview page in the Dell Server Management tab.

Misc Thoughts and Issues

• Hardware Provisioning and Deployment
  • Unfortunately, v1.0.1 cannot deploy a hypervisor to an internal SD card. This is how we plan to move forward with our ESXi installs (including the R610 I just ordered)
• Pricing
  • Retail pricing is $299.00 for up to 3 hosts, $799.00 for up to 10, $1,799.00 for up to 50 and $2,999.00 for up to 1000 hosts
    • If you have 1000 hosts, you can probably afford this. It might be hard to sell $800 to my management to manage my 5 hosts. Essentially, we'd have to save about 25 hours of work to break even
    • I'm not asking for it to be free. It does too much to be free and is really bordering on what you can define as a plug-in. What I'd like to see is up to 3 for free (throw the SMBs a bone and gain market share in the process), $300 for 5 hosts, etc.

Dell unveiled an update to 2 of their EqualLogic PS series array platforms today along with their first sub-$10k array. The new PS6100 and PS4100 series arrays are a refresh of their PS6000 and PS4000 units. The new boxes are being touted as having up to a 67% improvement in I/O performance. 

Speaking of the delegates, I want to thank Stephen Fosket and Matt Simmons for gathering such a great and diverse group of infrastructure professionals. Also, as one of the younger delegates, both in terms of age and experience, I want to thank the delegates for listening and taking the time to explore topics that were new for me.

I'm fortunate to be able to head straight to vacation right after TFD. Rather than head back to the crazy of work, I'll (hopefully) be able to better process what just happened in Austin. As I think back through the presentations and conversations, there's a lot of processing to do.

With demands growing much faster than our budget (centralized backup, Antivirus, patching, deployment, file and print services, CMS, LMS, better-than-just-pop-email), it was obvious that we could no longer afford physical servers. We had neither the budget nor the physical space, power, cooling, etc and had to come up with a better plan. Virtualization was the answer, and somebody had to do it. I fell in love with the technology and jumped right in. As most of you have probably experienced, it soon became the majority of my daily functions.

We quickly added one more ESX 3.5 host, consolidated 2 of the ESX 2.5 hosts into the 3.5 hosts, added a second shelf to the NetApp (now all of 3.5TB) and added a Dell PowerVault MD1000 attached to a PowerEdge 1950 running Red Hat serving as an NSF store (3TB also).

Sounds great. We should be set, right? Boy was I wrong. I had no idea how fast we could chew through storage and host resources. With our NetApp nearing End of Life (not to mention being well out of warranty), it was time to consider new storage and another host or 2. While we loved the performance of our NetApp, we couldn't afford a system with multiple controllers, couldn't afford death by licensed features and found it difficult to administer. Through a process I won't detail here, and with a price my Dell AE swore me me to protect, we decided to migrate to and standardize on EqualLogic. So we purchased a PS6000XV for primary storage (6.5TB usable) and a PS4000X for replication. 

We're now sitting with a single ESXi 4.1 cluster with 5 hosts and 3 EqualLogic arrays in two groups. We're still using the old NetApp iSCSI and MD1000 NFS SANs as tier 2 storage and now have a grand total of 26TB of storage (96TB more coming).

With the evolution of my workload and focus, as well as a new project building a remote data center in Houston as both a multi site cluster and DR site, I was offered the new position of Sr. Systems Administrator - Virtualization and Storage, which I gladly accepted. While this in part realigns my job title and description with what I actually do and where the Datacenter and IT services field is headed, it also adds more opportunities for growth. I will be taking on the role of Scrum Master (Srum is our internal project management framework), operate as lead/backup technician for the rest of the Sys Admin team and be responsible for server/service patch management oversight.

It's big and a little bit scary, but if im  not a little bit scared of what I'm doing, I get complacent and don't learn nearly as much.

Here's to being scared.