Cybereal’s Tech Blog
Rackspace Failure (update 3 of ?) (posted by Bob, 12/5/22)
Dear Cybereal Clients,
As of Monday at around 11am Rackspace is still being silent on this security issue and any possible timeframe for restoration of services. Communication to us partners has been non-existent with the exception of one email yesterday stating what seems to have been a tiny bit of good news, and that was the supposed ability to have affected email addresses temporarily forwarded to other email addresses. This can only be done via support ticket, so I immediately requested this setup for my secondary email account, and almost 24 hours that has still yet to be implemented. I have since submitted all forwarding requests from any clients that have requested them, and to the best of my knowledge none of them have been activated at the time of this posting.
As this outage stretches on and continues to impact our clients we will continue to reevaluate our suggestions, and are actively moving any clients over to the 365 platform that have request it. At this moment, our recommendations remain the same as what was provided to each client individually, but this will likely change unless there is a more comprehensive update released by Rackspace by the end of the day. So far, the only update from Rackspace aside from their System Status page (here), has been in the form of an e-mail from their CEO, which most people will not receive due to the outage. That e-mail was mainly just a cut-and-paste from an earlier update on their status page, and it provided no additional relevant information.
Thank you all again for your patience during this inexcusable failure from what was once a top-tier hosting provider..
-Bob
Rackspace Failure (update 2 of ?) (posted by Bob, 12/4/22)
Dear Cybereal Clients,
As this Rackspace outage stretches into its third day I wanted to get an additional general update posted so that we’re all on the same page. Though I have been in contact with everyone via text or alternative emails this update may contain some additional info we have not covered, and may assist you in deciding what path (ie, waiting it out, or migrating to 365) is best for your company. One quick note- Cybereal finished the conversion of our own email systems to Office 365 on Saturday and our email is back up and running. Though we are a small group, we have a very complex email environment, and the entire conversion took over six hours which should give you an idea of the effort involved in this process.
As we’ve mentioned, the general recommendation from Rackspace is that everyone move to 365 as soon as possible, and though that is likely the long-term plan at this point, doing so for most of our clients has an additional downside. The reason is, all of our clients still on Rackspace utilize the hybrid Hosted Exchange environment, where only certain mailboxes are Exchange (and effected by this issue), and all other mailboxes, usually the majority, are standard POP mailboxes and are currently unaffected. However, this conversion requires all mailboxes be converted to Office 365, so there will be additional downtime and disruption to those users as well. If we were dealing with any accounts that were strictly Exchange we would have already begun the transition.
Please see the below Q & A for additional info, and I will be in touch with each of you individually to discuss:
Q: How long do you anticipate this issue lasting?
A: Per Rackspace “We currently do not have an ETA for resolution. We are actively working with our support teams and anticipate our work may take several days”. Beyond that, this is almost impossible to answer, but my gut feeling says it will be resolved by late Tuesday, though admittedly that is just a hunch and not based on any data available to me, and it could be even longer than that.
Q: Should we transition to Office 365 as recommended by Rackspace?
A: This is a decision that will need to be made by you, the customer, but my suggestion as of the time of this writing is any clients with less than 30% of their users on Exchange wait this out. Whatever you decide we will put a plan in action to get you transitioned as quickly and as effectively as possible.
Q: Can we create a temporary email account elsewhere and forward the affected accounts to that address?
A: No. We have no control over the affected accounts including the ability to forward them, and Rackspace isn’t giving us partners any additional info or control. Plus, even though this doesn’t affect their POP mailboxes, any attempt I’ve made at creating additional accounts on that side have failed, so creating simple POP mailboxes for temporary communication isn’t an option.
Q: Will I lose email sent after 2:30am on Friday?
A: Per Rackspace “Possibly. We intend to update further as we get more information”. However, If you transition away from Rackspace Hosted Exchange, then yes, you definitely will lose any email that came in during that time. If you wait out this issue then you might still receive that email
Q: Rackspace is making this conversion to 365 sound easy, why are you saying it will take so long?
A: All Rackspace is assisting with is creating the Office 365 tenant account, adding mailboxes, and assigning licenses. That is the quickest and easiest part of the conversion but leaves clients with an empty mailbox and non-functional email client. The time consuming part of this process is backup up the existing email, reconfiguring the client (every email device needs to be reconfigured, including Outlook, scanners/copiers, applications that use email, smartphones, etc.), and restoring the backups.
Q: Would having a larger staff at Cybereal make the conversion to Office 365 quicker to accomplish?
A: That is unlikely. All MSPs must scale their customer base at a similar level in order to maintain profitability and stay in business. That means that any larger group would have more clients and a larger number of effected users. The number of staff they have available to assist would be at a similar ratio to what we offer, but we have the distinct advantage of experience and expertise, where the larger groups would be pulling low-level techs from all other areas to assist.
Thanks,
-Bob
Rackspace Failure (update 1 of ?) (posted by Bob, 12/3/22)
Dear Cybereal Customers,
At around 2:30am on Friday 12/2/23 one of our major email partners, Rackspace Technology, shut down their hosted Exchange environment due to a still-undisclosed security issue. This issue was only effecting their Exchange services and thankfully not all hosted e-mail. However, that still includes a significant number of Cybereal clients.
Around start of business on Friday these services were still unavailable and were impacting about 25% of our client base, including our own @cybereal.net email services. Steve and I did our best to keep everyone informed of this outage via text or phone call, and the expectation was that Rackspace would get this resolved within hours rather than days. However, late in the evening that same day, Rackspace finally released an update that included some information, and it was by all accounts worst-case scenario. Though we still do not know exactly what happened and what further impact this may have, Rackspace is indicating these services may not be back online for several days.
Though the hosted Exchange component is only effecting about 50 individual users within the Cybereal client-base, those are usually the business executives and sales people who rely most heavily on email, which is why they were setup on the more advanced and feature-rich hosted Exchange platform. Furthermore, due to the way Rackspace configured their hosting system, it is not currently possible to simply move an Exchange account to their standard POP mailbox and at least get those users back online for the time being. Instead, Rackspace is suggesting the unthinkable, and that is to move everyone to Office 365. While that will certainly be the long-term goal after this unacceptable outage, this requires moving not only the Exchange users to 365, but also the entire company’s e-mail system including even those users currently unaffected by this issue. While we have done this in the past, it has never been on a forced schedule and with so many clients at the same time. For this reason we are evaluating a plan of action based on the number of users impacted and coming up with a schedule.
I am not the most optimistic person (I prefer to think of myself as a realist!), but I still hope that Rackspace may get this sorted prior to the start of business on Monday, or at the very least may have some plan in place to use existing tools to accelerate the migration of their email services to Office 365. We have begun moving our own @cybereal.net e-mail to Office 365 and hope to have this in place by the end of the day on Saturday. In the meantime if you are one of the people effected by this incident and have an alternative email address you would like us to temporarily use for further communications please text Steve or I with that info and we will use that address for updates during this process.
Thank you very much for your patience during this ordeal.
Best,
Bob
Ahhh, the Classics! (posted by Bob, 9/18/18)
Hello fellow IT enthusiasts and Cybereal clients! Sorry for the lack of updates on this blog and to the website in general. I will try to make these updates a bit more frequent than once every 4 years or so…
Having said that, what has spurred me into action is the recent increase in spam and phishing as the result of a new virus and attack vector that is obtaining e-mail passwords. It is well known that there are dozens of tools available to recovered an email password that has been stored in Outlook, and it seems that a new virus or attack may be utilizing such a tool to obtain users email passwords. Once the password has been obtained, they setup a simple POP/SMTP account and send more viruses, spam, and phishing emails from that account, often as replies to existing emails you may have already received.
This method bypasses DMARC/SPF/DKIM and is very effective. Please be extra diligent regarding warnings from your virus software or redirects to websites you may not have intended to visit. Always contact your IT support person if you have even the slightest concern that your account or PC may have been compromised. Firewalls, antivirus, and monitoring can go a long way to protect your network and data, but good old fashioned diligence from the end user is still one of the best methods!
That brings me to this; I just received the below email to a temp account of mine.. I was almost nostalgic over the simplicity of this phish.. The good old days of a bygone era! For your enjoyment:
“My Compliments,
My name is Mr. Peter Adom, a politician and a previous member of Ghana’s executive committee on contract awards.
My purpose in connecting you is to crave your indulgence in assisting me to secure funds abroad. That was i kept to help me finance my senatorial elections campaign with the National Democratic Congress (NDC) on December 2016. Unfortunately, I lost the election.
But, I was able to deposit some kickback money from award of contracts to Chinese and Malaysian companies few years ago in a security vault in Ghana pending my finding of a trustworthy person to move out the funds from Ghana to an oversea bank account for business investment.
The need to contact you arose when the present elected government vowed to crack down on past government functionaries of the National Democratic Congress (NDC) which i am a member and tried to trace all the funds during our tenure in the office from the year 2008 to 2016. If they succeed in tracing this fund to me, they will seize it.
I wish to relocate this fund to a foreigner’s name to avoid any trace. All I need from you is an assurance that you can handle the amount involved (US$19,500,000) comfortably and that I can also trust you.
I will need the following information from you for onward execution of the aforementioned project.
1. Your full name:…………………….
2. Your Address:…………………. ….
3. Company name: …………………….
4. Position in Company:………………
5. Age:…………………….. …………
6. Nationality:……………… ………..
7. Telephone …………………………
8. Cell…………………….. ………….
9. Country of residence:…………….
Rest assured that there is no risk involved. Please inform me of your willingness to assist and co-operate with me, so that I can send you full details of this transaction.
Yours faithfully,
Mr. Peter Adom.”
CryptoWall 2.0 in the Wild (posted by Bob 10/17/14)
On Tuesday one of my clients was unlucky enough to contract the new Onion/TOR variant of the CryptoWall virus known as CryptoWall 2.0. This variant is pretty much undetectable at the moment, and even figuring out where it came in was a problem. Unlike other CryptoLocker/CryptoWall variants, this one does not pop up a screen telling people they are infected. I believe this may actually be a flaw in the design, rather than the way the system was intended to work, but either way it just adds to the amount of time the virus has to do its work before being interrupted.
Their fully updated antivirus solution did not detect the virus, nor did several others I tried. Only the old standby Malwarebytes was able to detect it during a heuristics scan, and flagged it as backdoor.bot. I removed the virus and sent it over to AVG for analysis. It would seem that Tuesday (10/14/15) was the launch day over here in the US, so it’s understandable that most antivirus programs did not have definitions that could detect this new variant, but hopefully that will be remedied soon. In the mean time, please make sure your backups are working properly and that your antivirus program is updating properly. If you’re unsure of your current antivirus or backup solutions, contact a Cybereal consultant who will be more than happy to discuss your options with you.
Windows 10 Announced (posted by Bob 10/2/14)
As I’ve been preaching to my clients and colleagues for a couple years now, I believe Windows 8 will be another of Microsoft’s skipped operating systems. Microsoft has a tendency to reach too far between OS deployments, and whatever they come up with is usually awful. Just as everyone skipped 2000 (which was actually a decent OS) and went from 98 to XP, and then skipped Vista and went to 7, they are now skipping 8 and will likely go to 10. Yes, ten. Not nine. Ten. I assume they opted to go with 10 because it is the closest copy of MacOS X (ten) to date, though some believe it is due to old code which references Windows 95/98 and may have caused issues.
Whatever the reason for the name, it looks as though they went back to their roots and just copied what works. From the brief tech demo I have seen and the images that are appearing online, the interface blends both aspects of 7 and 8 together, along with some new touches which I believe are inspired by the Mac. If you haven’t already seen it for yourself, shuffle on over to Microsoft’s blog to check it out. Unlike Apple’s use of IOS for iPhones and iPads (and now the Apple Watch), and MacOS X for it’s desktop hardware, Microsoft is still focusing on one OS to rule them all. It remains to be seen if one OS can truly provide a great interface for both desktop and mobile hardware, but Windows 10 looks to be making great strides. Assuming there are no egregious issues on the back end of the OS, Windows 10 looks to be shaping up to be a decent entry into the Windows portfolio- just remember, always wait until the first service pack has been issued before you deploy it in a business environment!